A method for high-quality wearable sensor data synthesis

Abstract

A high-quality wearable sensor data synthesis method relates to the technical field of data synthesis, and provides a generative model method for high-quality sensor data synthesis tasks, so that wearable sensor data samples with high information value can be synthesized according to the actual needs of the task. An active learning mechanism is introduced into adversarial training, and the information value of the sample is adaptively evaluated, so that the generative model can more effectively learn the distribution characteristics of data with high information value, thereby synthesizing wearable sensor data samples with more practical value.

Description

Technical Field

[0001] The present invention relates to the technical field of data synthesis, and in particular to a method for synthesizing high-quality wearable sensor data. Background Art

[0002] Wearable behavior recognition is an important research direction in the field of ubiquitous computing, and plays an important role in application scenarios such as health monitoring, auxiliary diagnosis and treatment, and smart home. The accuracy of wearable behavior recognition depends largely on the quantity and quality of data. However, due to the limitations of real-world conditions, such as inaccurate user movements, improper device wearing, environmental noise, sensor physical defects, labeling errors, and privacy protection restrictions, collecting and labeling a sufficient amount of sensor data is usually time-consuming, labor-intensive, and costly. Therefore, researchers use generative adversarial networks to synthesize wearable sensor data. However, existing methods cannot evaluate the information value of samples, and the models tend to synthesize a large number of ordinary samples and cannot efficiently synthesize samples with high information value. Therefore, the practical value of synthesized sensor data is limited. Summary of the invention

[0003] In order to overcome the shortcomings of the above technologies, the present invention provides a method for synthesizing wearable sensor data with high information value according to the actual needs of the task.

[0004] The technical solution adopted by the present invention to overcome the technical problems is:

[0005] A high-quality wearable sensor data synthesis method, comprising:

[0006] a) obtaining a set L of labeled wearable sensor data samples and a set U of unlabeled wearable sensor data samples;

[0007] b) Establish a total data sample set X′ based on the labeled wearable sensor data sample set L and the unlabeled wearable sensor data sample set U, The sensor data matrix stored in the i-th raw data storage file;

[0008] c) Perform data preprocessing on the total set of data samples X′ to obtain data samples S batch ;

[0009] d) The data sample S batch Input to the feature generator G f The output is the representation vector H of the wearable sensor sequence sample batch ;

[0010] e) The representation vector H of the wearable sensor sequence sample batch Input to the data generator Gd The synthesized wearable sensor time series segment data S is output b ' atch ;

[0011] f) The representation vector H of the wearable sensor sequence sample batch Input into the backbone network and output the vector Vector Input to feature discriminator D f In the output, the output vector Vector Input to feature classifier C f , the output vector is obtained

[0012] g) The data sample S batch Input to the data discriminator D d In the output, the output vector

[0013] h) Based on the synthesized wearable sensor time series segment data S b ' atch , output vector Output vector Output vector Train the feature generator G f , data generator G d , feature discriminator D f , feature classifier C f , data discriminator D d , and get the optimized feature generator G f ″″、Optimized data generator G d ″、Optimized feature discriminator D f ′、Optimized feature classifier C f ', optimized data discriminator D d ′;

[0014] i) The data sample S batch Input to the optimized feature generator G f ″″, optimized feature classifier C f ', the synthesized wearable sensor time series segment data S is output b ″ atch .

[0015] Further, step a) comprises the following steps:

[0016] a-1) The labeled wearable sensor data sample set L = {(l 1 ,y 1 ),(l 2,y 2 ),...,(l i ,y i ),…,(l M ,y M )},(l i ,y i ) is the i-th data-label pair, l i is the i-th labeled wearable sensor data, i∈{1,…,M}, M is the total number of labeled wearable sensor data, y i is the i-th labeled wearable sensor data l i The corresponding behavior category label, y i ∈{1,…,Y}, Y is the total number of behavior categories;

[0017] a-2) Unlabeled wearable sensor data sample collection is the i-th data-identifier pair, u i is the i-th unlabeled wearable sensor data, i∈{1,…,N}, N is the total number of unlabeled wearable sensor data, is the i-th unlabeled wearable sensor data u i The corresponding logo, Both are set to -1.

[0018] Further, step b) comprises the following steps:

[0019] b-1) Establishing a total set X of labeled wearable sensor data samples and unlabeled wearable sensor data samples, X = {L, U};

[0020] b-2) Establish a wearable sensor device set D, D = {d 1 ,d 2 ,…,d i ,…,d K}, d i is the i-th wearable sensor device, i∈{1,…,K}, K is the number of wearable sensor types, d i ={d i,1 ,d i,2 ,…,d i,a ,…,d i,A}, d i,a is the i-th wearable sensor device d i The ath dimension of , a∈{1,…,A}, A is the i-th wearable sensor device d i The number of dimensions in which data can be collected;

[0021] b-3) Establish a data storage file set F, F = {f 1,f 2 ,…,f i ,…,f K}, f i is the i-th original data storage file, i∈{1,…,W}, W is the number of original data storage files, and the i-th original data storage file f i The data stored is is a vector containing C columns of data collected by the sensor at time t, t∈{1,…,T}, T is the time series length of the collected data,

[0022] is the c-th column of data collected by the sensor at time t, c∈{1,…,C};

[0023] b-4) Store the i-th original data in file f i Stored Data Convert to Matrix Get the total set of data samples X′,

[0024] Further, step c) comprises the following steps:

[0025] c-1) Traverse W raw data storage files and check the i-th raw data storage file f i Stored Data The cth column of data collected by the sensor at time t Is there a missing value? If a column of data is detected to have a missing value, calculate the average value x of the previous and next valid values ​​of the column. mean , the average value x mean Replace missing values;

[0026] c-2) Calculate the i-th original data storage file f respectively i Stored Data The cth column of data collected by the sensor at time t Check the mean μ and standard deviation σ of column c Is there any data whose value is outside the interval [μ-3σ,μ+3σ]? If so, the value is determined to be abnormal data and the abnormal data is replaced with the mean μ;

[0027] c-3) According to the i-th behavior category label y i Filter out the corresponding labeled sensor data l from the W raw data storage files i, the filtered sensor data l i Perform Concat stacking operation on each row to obtain the data group L of the i-th category i , all data groups constitute a data set grouped by category {L 1 ,L 2 ,…,L i ,…,L Y}, each unlabeled sensor data u i Perform the Concat stacking operation to obtain the data set L -1 ;

[0028] c-4) Group the data set by category {L 1 ,L 2 ,…,L i ,...,L Y} Perform Concat stacking operations on rows to obtain annotated datasets reorganized by category

[0029] c-5) Reorganize the labeled dataset by category and dataset L -1 The i-th raw data is stored in file f i Stored Data The cth column of data collected by the sensor at time t Perform normalization operation to obtain a locally normalized labeled data set and dataset L -1 ′;

[0030] c-6) For the locally normalized labeled datasets and dataset L -1 ′Perform normalization operation to obtain a globally normalized labeled data set and dataset L -1 ″;

[0031] c-7) Use sliding window algorithm to analyze the labeled data set The i-th behavior category label y i The corresponding data is segmented into time series segments to obtain time series segments S i , using the sliding window algorithm to -1 The data in ″ is divided into time series segments to obtain time series segments S -1 ′, the M time series segments S i and timing fragments S -1 'Concat stacking operation is performed to obtain a complete set of time series segments S, the length of the sliding window L is set to seq_len = 100, and the sliding step size is set to step_size = 50;

[0032] c-8) Randomly select B time series segments from the complete time series segment set S to obtain a batch of data samples S batch .

[0033] Further, step d) comprises the following steps:

[0034] d-1) Data sample S batch The dimension is [B,L,C];

[0035] d-2) Feature Generator G f It consists of a spatial feature extraction module, a temporal feature extraction module, a spatiotemporal feature extraction module, a self-attention mechanism, a Reshape function, a first fully connected layer, a first ReLU function, a second fully connected layer, and a second ReLU function;

[0036] d-3) Feature Generator G f The spatial feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the spatial feature extraction module, and output the three-dimensional spatial features

[0037] d-4) Feature Generator G f The temporal feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Use the Reshape function to transform the dimension and obtain a data sample S with dimensions [B, C, L] batch , the data sample S with dimension [B,C,L] batch Input into the time feature extraction module, and output the three-dimensional time feature

[0038] d-5) Feature Generator G f The spatiotemporal feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Use the Reshape function to transform the dimension and obtain a data sample S with the dimension [B, 1, C, L] batch, the data sample S with dimension [B,1,C,L] batch Input into the spatiotemporal feature extraction module, and output the three-dimensional spatiotemporal features

[0039] d-6) The three-dimensional spatial features Three-dimensional temporal characteristics Three-dimensional space-time characteristics Perform the Concat stacking operation to obtain a three-dimensional feature vector The three-dimensional feature vector Input to the feature generator G f In the self-attention mechanism, the output is a three-dimensional feature vector The three-dimensional feature vector Using the feature generator G f The Reshape function performs dimension transformation to obtain a two-dimensional feature vector

[0040] d-7) The two-dimensional feature vector Input into the feature generator G in sequence f The first fully connected layer, the first ReLU function, the second fully connected layer, and the second ReLU function are used to output the representation vector H of the wearable sensor sequence sample. batch .

[0041] Further, step e) comprises the following steps:

[0042] e-1) Data generator G d It is composed of a first bidirectional long short-term memory network Bi-LSTM, a second bidirectional long short-term memory network Bi-LSTM, a first fully connected layer, a first ReLU function, a second fully connected layer, and a second ReLU function in sequence;

[0043] e-2) The representation vector H of the wearable sensor sequence sample batch Input into the data generator G in sequence d In the first bidirectional long short-term memory network Bi-LSTM and the second bidirectional long short-term memory network Bi-LSTM, the output is the intermediate layer output with time series characteristics The hidden dimension of the first bidirectional long short-term memory network Bi-LSTM and the second bidirectional long short-term memory network Bi-LSTM are both set to 100, and the dropout is set to 0.2;

[0044] e-3) Output the intermediate layer with time series features Input into the data generator G in sequence dThe first fully connected layer, the first ReLU function, the second fully connected layer, and the second ReLU function are used to output the synthesized wearable sensor time series segment data S b ' atch .

[0045] Further, step f) comprises the following steps:

[0046] f-1) The backbone network is composed of the first one-dimensional convolutional layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolutional layer, the second ReLU function, the second maximum pooling layer, the fully connected layer, and the third ReLU function in sequence;

[0047] f-2) The representation vector H of the wearable sensor sequence sample batch Input into the backbone network and output the vector

[0048] f-3) Feature Discriminator D f It is composed of the first fully connected layer, ReLU function, the second fully connected layer, and Sigmoid function. Input to feature discriminator D f In the output, the output vector is the feature graph vector of the i-th time series segment, The value is between [0,1]. When the value is 0, it is an unlabeled wearable sensor data sample. When the value is 1, it is an unlabeled wearable sensor data sample;

[0049] f-4) Feature classifier C f It is composed of the first fully connected layer, ReLU function, the second fully connected layer, and Sigmoid function. Input to feature classifier C f In the output, the output vector Output vector Represented as the representation vector H of the wearable sensor sequence sample batch is the behavior category label y i probability.

[0050] Further, step g) comprises the following steps:

[0051] g-1) Data Discriminator D d It consists of a time feature extraction module, a space feature extraction module, a first Reshape function, a time-space feature aggregation module, a second Reshape function, a fully connected layer, and a Sigmoid function;

[0052] g-2) Data Discriminator D d The temporal feature extraction module is composed of the first long short-term memory network LSTM, the first Dropout layer, the second long short-term memory network LSTM, the second Dropout layer, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the time feature extraction module, and output the time series features of the data

[0053] g-3) Data Discriminator D d The spatial feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the spatial feature extraction module, and output the spatial features of the data

[0054] g-4) The timing characteristics and spatial characteristics Perform Concat stacking operation to get vector Vector Using data discriminator D d The first Reshape function performs a dimension transformation operation to obtain a vector with dimensions [B, 1, C, L]

[0055] g-5) Data Discriminator D d The spatiotemporal feature aggregation module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, and the second maximum pooling layer. The vector with dimension [B, 1, C, L] Input into the spatiotemporal feature aggregation module and output the vector

[0056] g-6) vector Using data discriminator D d The second Reshape function performs the dimension transformation operation to obtain a two-dimensional space-time feature vector

[0057] g-7) The spatiotemporal feature vector Input into the data discriminator D in sequence d In the fully connected layer and Sigmoid function, the output vector is obtained is the feature graph vector of the i-th time series segment, The value is between [0,1]. When the value is 0, it is a real wearable sensor data sample. When the value is 1, it is a non-real wearable sensor data sample.

[0058] Further, step h) comprises the following steps:

[0059] h-1) Using the output vector Calculate the binary cross entropy loss function BCELoss and use the Adam optimizer to train the feature generator G using the binary cross entropy loss function BCELoss f , feature discriminator D f The model parameters are used to obtain the optimized feature generator G f ′ and the optimized feature discriminator D f ′, the learning rate of Adam optimizer is set to 0.001;

[0060] h-2) Using the output vector Calculate the cross entropy loss function and use the Adam optimizer to train the optimized feature generator G using the cross entropy loss function f ′, feature classifier C f , and get the optimized feature generator G f ″ and feature classifier C f ′, the learning rate of Adam optimizer is set to 0.001;

[0061] h-3) Construct an active learning mechanism and use the Least Confident Sampling Strategy to output vector From the time sequence segment S -1 ′Select the most informative samples, 20 at a time, and sequentially place the selected most informative samples from the time series segment S -1 ′ and added to the time sequence segment S i middle;

[0062] h-4) Repeat h-3) 100 times;

[0063] h-5) Using the output vector Calculate the binary cross entropy loss function BCELoss and use the Adam optimizer to train the feature generator G using the binary cross entropy loss function BCELoss f ″、Data generator G d , data discriminator D d The model parameters are used to obtain the optimized feature generator G f ″′, optimized data generator G d', optimized data discriminator D d ′, the learning rate of Adam optimizer is set to 0.001;

[0064] h-6) Calculate data sample S batch With the synthesized wearable sensor time series segment data S b ' atch The mean square error loss function MSELoss is used to train the feature generator G using the Adam optimizer using the mean square error loss function MSELoss f ″′ and data generator G d ′, get the optimized feature generator G f ″″、Optimized data generator G d ″.

[0065] Further, step i) comprises the following steps:

[0066] i-1) The data sample S batch Input to the optimized feature generator G f In ″″, the output is the representation vector H of the wearable sensor sequence sample batch ′;

[0067] i-2) The representation vector H of the wearable sensor sequence sample batch ′ is input into the backbone network and the output is vector

[0068] i-3) vector Input to the optimized feature classifier C f ′, the output vector is obtained Use the Least Confident Sampling Strategy by outputting the vector From the time sequence segment S -1 ′ Select n most informative samples and construct a high-value sample set S from the n most informative samples * ;

[0069] i-4) The high-value sample set S * Input to the optimized feature generator G f In ″″, the output is the representation vector H of the wearable sensor sequence sample batch ″;

[0070] i-5) The representation vector H of the wearable sensor sequence sample batch ″Input to the optimized data generator G d ″, the synthesized wearable sensor time series segment data S is output b ″atch .

[0071] The beneficial effects of the present invention are: providing a generative model method for high-quality sensor data synthesis tasks, so that wearable sensor data samples with high information value can be synthesized according to the actual needs of the task. The active learning mechanism is introduced into adversarial training, and the information value of the sample is adaptively evaluated, so that the generative model can more effectively learn the distribution characteristics of data with high information value, thereby synthesizing wearable sensor data samples with more practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a flow chart of the method of the present invention;

[0073] Figure 2 is the feature generator G of the present invention f Structure diagram of

[0074] Figure 3 The data generator G of the present invention d Structure diagram of

[0075] Figure 4 is the feature discriminator D of the present invention f and feature classifier C f Structure diagram of

[0076] Figure 5 The data discriminator D of the present invention d Structure diagram of . DETAILED DESCRIPTION

[0077] The following is combined with Figure 1 To Attachment Figure 5 The present invention is further described.

[0078] As attached Figure 1 As shown, a high-quality wearable sensor data synthesis method comprises:

[0079] a) Obtain a set L of labeled wearable sensor data samples and a set U of unlabeled wearable sensor data samples.

[0080] b) Establish a total data sample set X′ based on the labeled wearable sensor data sample set L and the unlabeled wearable sensor data sample set U, The sensor data matrix stored for the ith raw data storage file.

[0081] c) Perform data preprocessing on the total set of data samples X′ to obtain data samples S batch .

[0082] d) The data sample Sbatch Input to the feature generator G f The output is the representation vector H of the wearable sensor sequence sample batch .

[0083] e) The representation vector H of the wearable sensor sequence sample batch Input to the data generator G d The synthesized wearable sensor time series segment data S is output b ' atch .

[0084] f) The representation vector H of the wearable sensor sequence sample batch Input into the backbone network and output the vector Vector Input to feature discriminator D f In the output, the output vector Vector Input to feature classifier C f , the output vector is obtained

[0085] g) The data sample S batch Input to the data discriminator D d In the output, the output vector

[0086] h) Based on the synthesized wearable sensor time series segment data S b ' atch , output vector Output vector Output vector Train the feature generator G f , data generator G d , feature discriminator D f , feature classifier C f , data discriminator D d , and get the optimized feature generator G f ″″、Optimized data generator G d ″、Optimized feature discriminator D f ′、Optimized feature classifier C f ', optimized data discriminator D d ′.

[0087] i) The data sample S batch Input to the optimized feature generator G f ″″, optimized feature classifier C f ', the synthesized wearable sensor time series segment data S is output b ″ atch.

[0088] A generative model method for high-quality sensor data synthesis tasks is provided, so that wearable sensor data samples with high information value can be synthesized according to the actual needs of the task.

[0089] In one embodiment of the present invention, step a) comprises the following steps:

[0090] a-1) The labeled wearable sensor data sample set L = {(l 1 ,y 1 ),(l 2 ,y 2 ),…,(l i ,y i ),…,(l M ,y M )},(l i ,y i ) is the i-th data-label pair, l i is the i-th labeled wearable sensor data, i∈{1,...,M}, M is the total number of labeled wearable sensor data, y i is the i-th labeled wearable sensor data l i The corresponding behavior category label, y i ∈{1,...,Y}, Y is the total number of behavior categories.

[0091] a-2) Unlabeled wearable sensor data sample collection is the i-th data-identifier pair, u i is the i-th unlabeled wearable sensor data, i∈{1,...,N}, N is the total number of unlabeled wearable sensor data, is the i-th unlabeled wearable sensor data u i The corresponding logo, Both are set to -1.

[0092] In one embodiment of the present invention, step b) comprises the following steps:

[0093] b-1) Establish a total set X of labeled wearable sensor data samples and unlabeled wearable sensor data samples, X = {L, U}.

[0094] b-2) Establish a wearable sensor device set D, D = {d 1 ,d 2 ,…,d i ,...,d K}, d iis the i-th wearable sensor device, i∈{1,...,K}, K is the number of wearable sensor types, d i ={d i,1 ,d i,2 ,...,d i,a ,...,d i,A}, d i,a is the i-th wearable sensor device d i The ath dimension (axis or channel) of , a∈{1,...,A}, A is the i-th wearable sensor device d i The number of dimensions (axes or channels) in which data can be collected.

[0095] b-3) Establish a data storage file set F, F = {f 1 ,f 2 ,...,f i ,…,f K}, f i is the i-th original data storage file, i∈{1,…,W}, W is the number of original data storage files, and the i-th original data storage file f i The data stored is is a vector containing C columns of data collected by the sensor at time t, t∈{1,…,T}, T is the time series length of the collected data,

[0096] It is the c-th column (dimension, axis or channel) data collected by the sensor at time t, c∈{1,...,C}.

[0097] b-4) Store the i-th original data in file f i Stored Data Convert to Matrix Get the total set of data samples X′,

[0098] In one embodiment of the present invention, step c) comprises the following steps:

[0099] c-1) Traverse W raw data storage files and check the i-th raw data storage file f i Stored Data The cth column of data collected by the sensor at time t Is there a missing value? If a column of data is detected to have a missing value (expressed as NaN, i.e. non-numeric value), calculate the average value x of the previous and next valid values ​​of the column.mean , the average value x mean Replace missing values.

[0100] c-2) Calculate the i-th original data storage file f respectively i Stored Data The cth column of data collected by the sensor at time t Check the mean μ and standard deviation σ of column c Is there any data whose value is outside the interval [μ-3σ,μ+3σ]? If so, the value is determined to be abnormal data and the abnormal data is replaced by the mean μ.

[0101] c-3) According to the i-th behavior category label y i Filter out the corresponding labeled sensor data l from the W raw data storage files i , the filtered sensor data l i Perform Concat stacking operation on each row to obtain the data group L of the i-th category i , all data groups constitute a data set grouped by category {L 1 ,L 2 ,…,L i ,…,L Y}, each unlabeled sensor data u i Perform the Concat stacking operation to obtain the data set L -1 .

[0102] c-4) Group the data set by category {L 1 ,L 2 ,…,L i ,…,L Y} Perform Concat stacking operations on rows to obtain annotated datasets reorganized by category

[0103] c-5) Reorganize the labeled dataset by category and dataset L -1 The i-th raw data is stored in file f i Stored Data The cth column of data collected by the sensor at time t Perform normalization operation to obtain a locally normalized labeled data set and dataset L -1 ′.

[0104] c-6) For the locally normalized labeled datasets and dataset L -1 ′Perform normalization operation to obtain a globally normalized labeled data set and dataset L -1 ″.

[0105] c-7) Use sliding window algorithm to analyze the labeled data set The i-th behavior category label y i The corresponding data is segmented into time series segments to obtain time series segments S i , using the sliding window algorithm to -1 The data in ″ is divided into time series segments to obtain time series segments S -1 ′, the M time series segments S i and timing fragments S -1 ′Concat stacking operation is performed to obtain a complete set of time series segments S, the length L of the sliding window is set to seq_len=100, and the sliding step size is set to step_size=50.

[0106] c-8) Randomly select B time series segments from the complete time series segment set S to obtain a batch of data samples S batch .

[0107] As attached Figure 2 As shown, in one embodiment of the present invention, step d) comprises the following steps:

[0108] d-1) Data sample S batch The dimension is [B,L,C].

[0109] d-2) Feature Generator G f It consists of a spatial feature extraction module, a temporal feature extraction module, a spatiotemporal feature extraction module, a self-attention mechanism, a Reshape function, a first fully connected layer, a first ReLU function, a second fully connected layer, and a second ReLU function. Feature generator G f The main function is to map the labeled wearable sensor data sample set L and the unlabeled wearable sensor data sample set U into a unified feature space, and extract potential features that can effectively characterize the wearable sensor data samples.

[0110] d-3) Feature Generator G f The spatial feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the spatial feature extraction module, and output the three-dimensional spatial features

[0111] d-4) Feature Generator G fThe temporal feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Use the Reshape function to transform the dimension and obtain a data sample S with dimensions [B, C, L] batch , the data sample S with dimension [B,C,L] batch Input into the time feature extraction module, and output the three-dimensional time feature

[0112] d-5) Feature Generator G f The spatiotemporal feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Use the Reshape function to transform the dimension and obtain a data sample S with the dimension [B, 1, C, L] batch , the data sample S with dimension [B,1,C,L] batch Input into the spatiotemporal feature extraction module, and output the three-dimensional spatiotemporal features

[0113] d-6) The three-dimensional spatial features Three-dimensional temporal characteristics Three-dimensional space-time characteristics Perform the Concat stacking operation to obtain a three-dimensional feature vector The three-dimensional feature vector Input to the feature generator G f In the self-attention mechanism, the output is a three-dimensional feature vector The three-dimensional feature vector Using the feature generator G f The Reshape function performs dimension transformation to obtain a two-dimensional feature vector

[0114] d-7) The two-dimensional feature vector Input into the feature generator G in sequence f The first fully connected layer, the first ReLU function, the second fully connected layer, and the second ReLU function are used to output the representation vector H of the wearable sensor sequence sample. batch .

[0115] As attached Figure 3As shown, in one embodiment of the present invention, step e) comprises the following steps:

[0116] e-1) Data generator G d It is composed of a first bidirectional long short-term memory network Bi-LSTM, a second bidirectional long short-term memory network Bi-LSTM, a first fully connected layer, a first ReLU function, a second fully connected layer, and a second ReLU function.

[0117] e-2) The representation vector H of the wearable sensor sequence sample batch Input into the data generator G in sequence d In the first bidirectional long short-term memory network Bi-LSTM and the second bidirectional long short-term memory network Bi-LSTM, the output is the intermediate layer output with time series characteristics The hidden layer dimensions of the first bidirectional long short-term memory network Bi-LSTM and the second bidirectional long short-term memory network Bi-LSTM are both set to 100, and the dropout is set to 0.2.

[0118] e-3) Output the intermediate layer with time series features Input into the data generator G in sequence d The first fully connected layer, the first ReLU function, the second fully connected layer, and the second ReLU function are used to output the synthesized wearable sensor time series segment data S b ' atch .

[0119] As attached Figure 4 As shown, in one embodiment of the present invention, step f) comprises the following steps:

[0120] f-1) The backbone network is composed of the first one-dimensional convolutional layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolutional layer, the second ReLU function, the second maximum pooling layer, the fully connected layer, and the third ReLU function.

[0121] f-2) The representation vector H of the wearable sensor sequence sample batch Input into the backbone network and output the vector

[0122] f-3) Feature Discriminator D f It is composed of the first fully connected layer, ReLU function, the second fully connected layer, and Sigmoid function. Input to feature discriminator D f In the output, the output vector is the feature graph vector of the i-th time series segment, The value is between [0,1]. When the value is 0, it is an unlabeled wearable sensor data sample. When the value is 1, it is an unlabeled wearable sensor data sample. Feature discriminator D f Mainly used to characterize the vector H of wearable sensor sequence samples batch The corresponding wearable sensor data samples are identified in the states with and without category labels.

[0123] f-4) Feature classifier C f It is composed of the first fully connected layer, ReLU function, the second fully connected layer, and Sigmoid function. Input to feature classifier C f In the output, the output vector Output vector Represented as the representation vector H of the wearable sensor sequence sample batch is the behavior category label y i The probability of feature classifier C f Mainly used to characterize the vector H of wearable sensor sequence samples batch The corresponding wearable sensor data sample category is predicted.

[0124] As attached Figure 5 As shown, in one embodiment of the present invention, step g) comprises the following steps:

[0125] g-1) Data discriminator D d It consists of a time feature extraction module, a space feature extraction module, a first Reshape function, a spatiotemporal feature aggregation module, a second Reshape function, a fully connected layer, and a Sigmoid function. Data discriminator D d Mainly used to judge the input data sample S batch Whether it is real data, if the output is 0, it means that the input wearable sensor sample is recognized as synthetic data, and if the output is 1, it means that the input wearable sensor sample is recognized as real data.

[0126] g-2) Data Discriminator D d The temporal feature extraction module is composed of the first long short-term memory network LSTM, the first Dropout layer, the second long short-term memory network LSTM, the second Dropout layer, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the time feature extraction module, and output the time series features of the data

[0127] g-3) Data Discriminator D dThe spatial feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the spatial feature extraction module, and output the spatial features of the data

[0128] g-4) The timing characteristics and spatial characteristics Perform Concat stacking operation to get vector Vector Using data discriminator D d The first Reshape function performs a dimension transformation operation to obtain a vector with dimensions [B, 1, C, L]

[0129] g-5) Data Discriminator D d The spatiotemporal feature aggregation module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, and the second maximum pooling layer. The vector with dimension [B, 1, C, L] Input into the spatiotemporal feature aggregation module and output the vector

[0130] g-6) vector Using data discriminator D d The second Reshape function performs the dimension transformation operation to obtain a two-dimensional space-time feature vector

[0131] g-7) The spatiotemporal feature vector Input into the data discriminator D in sequence d In the fully connected layer and Sigmoid function, the output vector is obtained is the feature graph vector of the i-th time series segment, The value is between [0,1]. When the value is 0, it is a real wearable sensor data sample. When the value is 1, it is a non-real wearable sensor data sample.

[0132] In one embodiment of the present invention, step h) comprises the following steps:

[0133] h-1) Using the output vector Calculate the binary cross entropy loss function BCELoss and use the Adam optimizer to train the feature generator G using the binary cross entropy loss function BCELoss f , feature discriminator D f The model parameters are used to obtain the optimized feature generator G f ′ and the optimized feature discriminator D f ′, the learning rate of Adam optimizer is set to 0.001.

[0134] h-2) Using the output vector Calculate the cross entropy loss function and use the Adam optimizer to train the optimized feature generator G using the cross entropy loss function f ′, feature classifier C f , and get the optimized feature generator G f ″ and feature classifier C f ′, the learning rate of Adam optimizer is set to 0.001.

[0135] h-3) Construct an active learning mechanism and use the LeastConfident Sampling Strategy to output vector From the time sequence segment S -1 ′Select the most informative samples, 20 at a time, and sequentially place the selected most informative samples from the time series segment S -1 ′ and added to the time sequence segment S i middle.

[0136] h-4) Repeat h-3) 100 times to complete the consistent representation training. The goal is to make the feature generator G f , feature discriminator D f , feature classifier C f The three can work together to make the data set L -1 "It is fully and effectively utilized.

[0137] h-5) Using the output vector Calculate the binary cross entropy loss function BCELoss and use the Adam optimizer to train the feature generator G using the binary cross entropy loss function BCELoss f ″、Data generator G d , data discriminator D d The model parameters are used to obtain the optimized feature generator G f ″′, optimized data generator G d ', optimized data discriminator D d′, the learning rate of the Adam optimizer is set to 0.001. The adversarial training is completed to make the feature generator G f , data generator G d , data discriminator D d The three can work together to finally synthesize a data discriminator G d Differentiated realistic wearable sensor data.

[0138] h-6) Calculate data sample S batch With the synthesized wearable sensor time series segment data S b ' atch The mean square error loss function MSELoss is used to train the feature generator G using the Adam optimizer using the mean square error loss function MSELoss f ″′ and data generator G d ′, get the optimized feature generator G f ″″、Optimized data generator G d ″. Complete the reconstruction training and strengthen the feature generator G f and data generator G d The ability to capture the intrinsic characteristics and local details of wearable sensor data while reducing the pattern collapse phenomenon in the generation process.

[0139] In one embodiment of the present invention, step i) comprises the following steps:

[0140] i-1) The data sample S batch Input to the optimized feature generator G f In ″″, the output is the representation vector H of the wearable sensor sequence sample batch ′.

[0141] i-2) The representation vector H of the wearable sensor sequence sample batch ′ is input into the backbone network and the output is vector

[0142] i-3) vector Input to the optimized feature classifier C f ′, the output vector is obtained Use the Least Confident Sampling Strategy by outputting the vector From the time sequence segment S -1 ′ Select n most informative samples and construct a high-value sample set S from the n most informative samples * .

[0143] i-4) The high-value sample set S *Input to the optimized feature generator G f In ″″, the output is the representation vector H of the wearable sensor sequence sample batch ″.

[0144] i-5) The representation vector H of the wearable sensor sequence sample batch ″Input to the optimized data generator G d ″, the synthesized wearable sensor time series segment data S of high-value information is output b ″ atch . By the optimized feature generator G f ″″ and optimized data generator G d "Working together, they can generate a specified number of high-information-value wearable sensor time-series data samples.

[0145] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for synthesizing high-quality wearable sensor data, characterized in that: include: a) obtaining a set L of labeled wearable sensor data samples and a set U of unlabeled wearable sensor data samples; b) Establish a total data sample set X′ based on the labeled wearable sensor data sample set L and the unlabeled wearable sensor data sample set U, The sensor data matrix stored in the i-th raw data storage file; c) Perform data preprocessing on the total set of data samples X′ to obtain data samples S batch ; d) The data sample S batch Input to the feature generator G f The output is the representation vector H of the wearable sensor sequence sample batch ; e) The representation vector H of the wearable sensor sequence sample batch Input to the data generator G d The synthesized wearable sensor time series segment data S′ is output batch ; f) The representation vector H of the wearable sensor sequence sample batch Input into the backbone network and output the vector Vector Input to feature discriminator D f In the output, the output vector Vector Input to feature classifier C f , the output vector is obtained g) The data sample S batch Input to the data discriminator D d In the output, the output vector h) Based on the synthesized wearable sensor time series segment data S b ' atch , output vector Output vector Output vector Train the feature generator G f , data generator G d , feature discriminator D f , feature classifier C f , data discriminator D d , and get the optimized feature generator G f ″″、Optimized data generator G d ″、Optimized feature discriminator D f ′、Optimized feature classifier C f ', optimized data discriminator D d ′; i) The data sample S batch Input to the optimized feature generator G f ″″, optimized feature classifier C f ', the synthesized wearable sensor time series segment data S is output b ″ atch .

2. The high-quality wearable sensor data synthesis method according to claim 1, characterized in that: Step a) comprises the following steps: a-1) The labeled wearable sensor data sample set L = {(l1,y1),(l2,y2),…,(l i ,y i ),…,(l M ,y M )},(l i ,y i ) is the i-th data-label pair, l i is the i-th labeled wearable sensor data, i∈{1,…,M}, M is the total number of labeled wearable sensor data, y i is the i-th labeled wearable sensor data l i The corresponding behavior category label, y i ∈{1,…,Y}, Y is the total number of behavior categories; a-2) Unlabeled wearable sensor data sample collection is the i-th data-identifier pair, u i is the i-th unlabeled wearable sensor data, i∈{1,…,N}, N is the total number of unlabeled wearable sensor data, is the i-th unlabeled wearable sensor data u i The corresponding logo, Both are set to -1.

3. The high-quality wearable sensor data synthesis method according to claim 2, characterized in that: Step b) comprises the following steps: b-1) Establishing a total set X of labeled wearable sensor data samples and unlabeled wearable sensor data samples, X = {L, U}; b-2) Establish a wearable sensor device set D, D = {d1, d2, ..., d i ,…,d K }, d i is the i-th wearable sensor device, i∈{1,…,K}, K is the number of wearable sensor types, d i ={d i,1 ,d i,2 ,…,d i,a ,…,d i,A }, d i,a is the i-th wearable sensor device d i The ath dimension of , a∈{1,…,A}, A is the i-th wearable sensor device d i The number of dimensions in which data can be collected; b-3) Establish a data storage file set F, F = {f1, f2, ..., f i ,...,f K }, f i is the i-th original data storage file, i∈{1,...,W}, W is the number of original data storage files, and the i-th original data storage file f i The data stored is is a vector containing C columns of data collected by the sensor at time t, t∈{1,...,T}, T is the time series length of the collected data, is the c-th column of data collected by the sensor at time t, c∈{1,...,C}; b-4) Store the i-th original data in file f i Stored Data Convert to Matrix Get the total set of data samples X′, 4. The high-quality wearable sensor data synthesis method according to claim 3, characterized in that: Step c) comprises the following steps: c-1) Traverse W raw data storage files and check the i-th raw data storage file f i Stored Data The cth column of data collected by the sensor at time t Is there a missing value? If a column of data is detected to have a missing value, calculate the average value x of the previous and next valid values ​​of the column. mean , the average value x mean Replace missing values; c-2) Calculate the i-th original data storage file f respectively i Stored Data The cth column of data collected by the sensor at time t Check the mean μ and standard deviation σ of column c Is there any data whose value is outside the interval [μ-3σ,μ+3σ]? If so, the value is determined to be abnormal data and the abnormal data is replaced with the mean μ; c-3) According to the i-th behavior category label y i Filter out the corresponding labeled sensor data l from the W raw data storage files i , the filtered sensor data l i Perform Concat stacking operation on each row to obtain the data group L of the i-th category i , all data groups constitute a data set grouped by category {L1,L2,…,L i ,…,L Y }, each unlabeled sensor data u i Perform the Concat stacking operation to obtain the data set L -1 ; c-4) The data set {L1, L2, …, L i ,...,L Y } Perform Concat stacking operations on rows to obtain annotated datasets reorganized by category c-5) Reorganize the labeled dataset by category and dataset L -1 The i-th raw data is stored in file f i Stored Data The cth column of data collected by the sensor at time t Perform normalization operation to obtain a locally normalized labeled data set and dataset L -1 ′; c-6) respectively for the locally normalized labeled datasets and dataset L -1 ′Perform normalization operation to obtain a globally normalized labeled data set and dataset L -1 ″; c-7) Use sliding window algorithm to analyze the labeled data set The i-th behavior category label y i The corresponding data is segmented into time series segments to obtain time series segments S i , using the sliding window algorithm to -1 The data in ″ is divided into time series segments to obtain time series segments S -1 ′, M time series segments S i and timing fragments S -1 'Concat stacking operation is performed to obtain a complete set of time series segments S, the length of the sliding window L is set to seq_len = 100, and the sliding step size is set to step_size = 50; c-8) Randomly select B time series segments from the complete time series segment set S to obtain a batch of data samples S batch .

5. The high-quality wearable sensor data synthesis method according to claim 4, characterized in that: Step d) comprises the following steps: d-1) Data sample S batch The dimension is [B,L,C]; d-2) Feature Generator G f It consists of a spatial feature extraction module, a temporal feature extraction module, a spatiotemporal feature extraction module, a self-attention mechanism, a Reshape function, a first fully connected layer, a first ReLU function, a second fully connected layer, and a second ReLU function; d-3) Feature Generator G f The spatial feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the spatial feature extraction module, and output the three-dimensional spatial features d-4) Feature Generator G f The temporal feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Use the Reshape function to transform the dimension and obtain a data sample S with dimensions [B, C, L] batch , the data sample S with dimension [B,C,L] batch Input into the time feature extraction module, and output the three-dimensional time feature d-5) Feature Generator G f The spatiotemporal feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, the Reshape function, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Use the Reshape function to transform the dimension and obtain a data sample S with the dimension [B, 1, C, L] batch , the data sample S with dimension [B,1,C,L] batch Input into the spatiotemporal feature extraction module, and output the three-dimensional spatiotemporal features d-6) The three-dimensional spatial features Three-dimensional temporal characteristics Three-dimensional space-time characteristics Perform Concat stacking operation to obtain a three-dimensional feature vector The three-dimensional feature vector Input to the feature generator G f In the self-attention mechanism, the output is a three-dimensional feature vector The three-dimensional feature vector Using the feature generator G f The Reshape function performs dimension transformation to obtain a two-dimensional feature vector d-7) The two-dimensional feature vector Input into the feature generator G in sequence f The first fully connected layer, the first ReLU function, the second fully connected layer, and the second ReLU function are used to output the representation vector H of the wearable sensor sequence sample. batch .

6. The high-quality wearable sensor data synthesis method according to claim 1, characterized in that: Step e) comprises the following steps: e-1) Data generator G d It is composed of a first bidirectional long short-term memory network Bi-LSTM, a second bidirectional long short-term memory network Bi-LSTM, a first fully connected layer, a first ReLU function, a second fully connected layer, and a second ReLU function in sequence; e-2) The representation vector H of the wearable sensor sequence sample batch Input into the data generator G in sequence d In the first bidirectional long short-term memory network Bi-LSTM and the second bidirectional long short-term memory network Bi-LSTM, the output is the intermediate layer output with time series characteristics The hidden dimension of the first bidirectional long short-term memory network Bi-LSTM and the second bidirectional long short-term memory network Bi-LSTM are both set to 100, and the dropout is set to 0.2; e-3) Output the intermediate layer with time series features Input into the data generator G in sequence d The first fully connected layer, the first ReLU function, the second fully connected layer, and the second ReLU function are used to output the synthesized wearable sensor time series segment data S b ' atch .

7. The high-quality wearable sensor data synthesis method according to claim 4, characterized in that: Step f) comprises the following steps: f-1) The backbone network is composed of the first one-dimensional convolutional layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolutional layer, the second ReLU function, the second maximum pooling layer, the fully connected layer, and the third ReLU function in sequence; f-2) The representation vector H of the wearable sensor sequence sample batch Input into the backbone network and output the vector f-3) Feature Discriminator D f It is composed of the first fully connected layer, ReLU function, the second fully connected layer, and Sigmoid function. Input to feature discriminator D f In the output, the output vector is the feature graph vector of the i-th time series segment, The value is between [0,1]. When the value is 0, it is an unlabeled wearable sensor data sample. When the value is 1, it is an unlabeled wearable sensor data sample; f-4) Feature classifier C f It is composed of the first fully connected layer, ReLU function, the second fully connected layer, and Sigmoid function. Input to feature classifier C f In the output, the output vector Output vector Represented as the representation vector H of the wearable sensor sequence sample batch is the behavior category label y i probability.

8. The high-quality wearable sensor data synthesis method according to claim 1, characterized in that: Step g) comprises the following steps: g-1) Data Discriminator D d It consists of a time feature extraction module, a space feature extraction module, a first Reshape function, a time-space feature aggregation module, a second Reshape function, a fully connected layer, and a Sigmoid function; g-2) Data Discriminator D d The temporal feature extraction module is composed of the first long short-term memory network LSTM, the first Dropout layer, the second long short-term memory network LSTM, the second Dropout layer, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the time feature extraction module, and output the time series features of the data g-3) Data Discriminator D d The spatial feature extraction module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, the second maximum pooling layer, and the fully connected layer. The data sample S with dimension [B, L, C] is batch Input into the spatial feature extraction module, and output the spatial features of the data g-4) The timing characteristics and spatial characteristics Perform Concat stacking operation to get vector Vector Using data discriminator D d The first Reshape function performs a dimension transformation operation to obtain a vector with dimensions [B, 1, C, L] g-5) Data Discriminator D d The spatiotemporal feature aggregation module is composed of the first one-dimensional convolution layer, the first ReLU function, the first maximum pooling layer, the second one-dimensional convolution layer, the second ReLU function, and the second maximum pooling layer, which transforms the vector with dimension [B, 1, C, L] Input into the spatiotemporal feature aggregation module and output the vector g-6) vector Using data discriminator D d The second Reshape function performs the dimension transformation operation to obtain a two-dimensional space-time feature vector g-7) The spatiotemporal feature vector Input into the data discriminator D in sequence d In the fully connected layer and Sigmoid function, the output vector is obtained is the feature graph vector of the i-th time series segment, The value is between [0,1]. When the value is 0, it is a real wearable sensor data sample. When the value is 1, it is a non-real wearable sensor data sample.

9. The high-quality wearable sensor data synthesis method according to claim 1, characterized in that: Step h) comprises the following steps: h-1) Using the output vector Calculate the binary cross entropy loss function BCELoss and use the Adam optimizer to train the feature generator G using the binary cross entropy loss function BCELoss f , feature discriminator D f The model parameters are used to obtain the optimized feature generator G f ′ and the optimized feature discriminator D f ′, the learning rate of Adam optimizer is set to 0.001; h-2) Using the output vector Calculate the cross entropy loss function and use the Adam optimizer to train the optimized feature generator G using the cross entropy loss function f ′, feature classifier C f , and get the optimized feature generator G f ″ and feature classifier C f ′, the learning rate of Adam optimizer is set to 0.001; h-3) Construct an active learning mechanism and use the Least Confident Sampling Strategy to output vector From the time sequence segment S -1 ′Select the most informative samples, 20 at a time, and sequentially place the selected most informative samples from the time series segment S -1 ′ and added to the time sequence segment S i middle; h-4) Repeat h-3) 100 times; h-5) Using the output vector Calculate the binary cross entropy loss function BCELoss and use the Adam optimizer to train the feature generator G using the binary cross entropy loss function BCELoss f ″、Data generator G d , data discriminator D d The model parameters are used to obtain the optimized feature generator G f ″′, optimized data generator G d ', optimized data discriminator D d ′, the learning rate of Adam optimizer is set to 0.001; h-6) Calculate data sample S batch With the synthesized wearable sensor time series segment data S b ' atch The mean square error loss function MSELoss is used to train the feature generator G using the Adam optimizer using the mean square error loss function MSELoss f ″′ and data generator G d ′, get the optimized feature generator G f ″″、Optimized data generator G d ″.

10. The high-quality wearable sensor data synthesis method according to claim 9, characterized in that: Step i) comprises the following steps: i-1) The data sample S batch Input to the optimized feature generator G f In ″″, the representation vector H of the wearable sensor sequence sample is output batch ′; i-2) The representation vector H of the wearable sensor sequence sample batch ′ is input into the backbone network and the output is vector i-3) vector Input to the optimized feature classifier C f ′, the output vector is obtained Use the Least Confident Sampling Strategy by outputting the vector From the time sequence segment S -1 ′ Select n most informative samples and construct a high-value sample set S from the n most informative samples * ; i-4) The high-value sample set S * Input to the optimized feature generator G f In ″″, the representation vector H of the wearable sensor sequence sample is output batch ″; i-5) The representation vector H of the wearable sensor sequence sample batch ″Input to the optimized data generator G d ″, the synthesized wearable sensor time series segment data S is output b ″ atch .

Images