Human body activity identification method and system based on semi-supervised deep learning
By adopting the semi-supervised deep learning method ACS-HAR in the human activity recognition model, using dynamic confidence adjustment and focus loss, combined with partial convolution and bidirectional GRU, the problem of dependence on labeled data in the existing technology is solved, and the recognition is improved Accuracy and generalization capabilities reduce data collection costs.
Patent Information
- Application Number
- CN202410972855.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
AI Technical Summary
Existing human activity recognition models require a large amount of annotated data for training, which leads to high cost and inefficiency in data collection, especially in feature extraction and model training, which requires a lot of manual intervention and expert knowledge.
ACS-HAR is used to optimize the model's generalization ability by applying different degrees of perturbation to unlabeled samples, and introduce focus loss and dynamic confidence adjustment mechanisms, combining partial convolution and bidirectional GRU to optimize the model's calculations. Cost and memory access frequency.
It significantly improves the accuracy and generalization ability of human activity recognition, reduces the dependence on labeled data, reduces the amount of labeled data required for model training, and effectively solves the class imbalance problem caused by pseudo-labels.
Smart Images

Figure CN118916741A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and relates to a human activity recognition method and system based on semi-supervised deep learning. Background Art
[0002] In recent years, the use of smart devices and Internet of Things (IoT) devices has increased the importance of human activity recognition in various scenarios, such as smart home systems, healthcare, and health monitoring. HAR is used to determine the user's activities based on sensor signals. HAR can be divided into two categories: wearable sensor-based methods and environmental sensing methods. Wearable sensor-based HAR methods have been widely used due to their extensive information content, seamless device integration, and stronger privacy protection, and have dominated daily activity monitoring. However, extracting rich semantic representations from activity signals remains a key challenge.
[0003] Current HAR methods can be mainly divided into two categories: feature engineering-based methods and deep learning-based methods. Feature engineering-based methods generally use traditional machine learning methods to identify different activities. These methods involve defining and extracting various features from sensor signals, such as statistical features or frequency characteristics. These extracted features usually need to be carefully selected in order for traditional machine learning methods to achieve relatively high accuracy. However, these methods require expert knowledge and manual intervention in the feature extraction and selection process, and may still not achieve optimal performance.
[0004] In order to overcome the limitations of traditional hand-crafted feature methods, some deep learning-based methods have been proposed. Common deep learning models for HAR are recurrent neural networks (RNNs) and convolutional neural networks (CNNs). RNNs are good at analyzing time series data and capturing a wide range of temporal dependencies. On the other hand, CNNs are good at extracting features from signals, showing a high degree of invariance to spatial or local changes and not affected by spatiotemporal background. Combining CNN with RNN improves the accuracy of the model by combining spatial and temporal information from sensors, resulting in a more robust feature representation. In addition, attention mechanisms are introduced to improve the effectiveness of the model. However, these methods require a large amount of labeled data to achieve satisfactory activity recognition accuracy. Collecting a large amount of labeled data is labor-intensive and even costly, which limits the practical application of such HAR methods.
[0005] In order to reduce the cost of collecting labeled data, some researchers have turned to semi-supervised learning methods to strengthen model training. Semi-supervised learning is a machine learning paradigm that combines a large amount of unlabeled data with a small amount of labeled data. Typical semi-supervised learning algorithms are mainly divided into pseudo-label-based methods and consistency regularization methods. Pseudo-label-based methods first use labeled data to train weak classifiers, and then predict unlabeled data to generate pseudo-labels. This type of method requires setting a confidence standard and selecting unlabeled samples with high confidence to continuously expand the training set.
[0006] In contrast, consistency regularization methods encourage the model to produce consistent predictions for the same input under different data augmentation conditions, thereby learning the intrinsic invariance of the data and improving the model's robustness to input changes. However, these methods usually require setting higher confidence standards to filter unlabeled samples to improve model performance. But in the early stages of training, too high confidence standards may lead to low data utilization and limit the learning efficiency of the model. At the same time, too high confidence standards may also lead to class imbalance of pseudo-labels, causing cognitive bias in the model, and ultimately affecting the overall performance of the model. Summary of the invention
[0007] In view of this, the purpose of the present invention is to provide a method and system for human activity recognition based on semi-supervised deep learning, aiming to solve the problem that the existing human activity recognition model requires a large amount of labeled data to train the model, and collecting a large amount of labeled activity data usually requires a lot of manpower and is even costly. The semi-supervised learning method ACS-HAR proposed in the present invention with both high precision and robustness adopts a semi-supervised learning framework and enhances the training process with unlabeled samples. Specifically, different degrees of perturbations are applied to the same unlabeled sample to prompt the model to produce consistent prediction results under these perturbations, thereby improving the generalization ability of the model; at the same time, the confidence standard is adaptively adjusted according to the learning state of the model, and high-confidence unlabeled samples are effectively selected for training; in order to avoid the class imbalance problem caused by pseudo-labels, focal loss is introduced; finally, partial convolution and bidirectional GRU are combined to capture the spatial and temporal dependencies of samples, thereby reducing the computational cost and memory access frequency of the model. Experimental results on four mainstream HAR datasets confirm that the method of the present invention is superior to existing semi-supervised HAR methods, especially in terms of accuracy and generalization ability.
[0008] The specific technical solutions of the present invention are as follows:
[0009] A semi-supervised deep learning method for human activity recognition, the method comprising the following steps: S1, supervised training of small tag sensor signals in smart devices; S2, using a trained model to generate pseudo labels for unlabeled data; S3, iteratively selecting pseudo label samples with high confidence and incorporating them into a training set until all high-quality samples have been used; S4, using the trained model to recognize human activities.
[0010] Further, in step S1, during the supervised training phase of the small tag sensor signal, the processed data set From the inertial sensors built into smart devices, including accelerometers, gyroscopes, and magnetometers; during supervised training, sensor signals It is input into the supervision branch for feature encoding; the supervision branch consists of a spatial feature encoder, a temporal feature encoder, and a classifier. This structure enables the model to deeply learn the key features of sensor data in spatial and temporal dimensions; then, these learned features are mapped to specific behavior categories through the top classifier, where and denote the labeled sample set and the unlabeled sample set respectively, Represents the sensor signal, Indicates the related activity category.
[0011] Further, in step S2, the trained model is used to generate pseudo labels for the unlabeled data, including implementing two different enhancement strategies for the unlabeled samples: weak enhancement (such as random scaling) and strong enhancement (combining time variation and random scaling). These enhanced unlabeled samples It is then input into the trained supervision branch to generate pseudo labels.
[0012] Further, in step S3, the iterative selection of high-confidence pseudo-label samples and inclusion in the training set until all high-quality samples are used up includes: using an adaptive confidence adjustment mechanism to dynamically adjust the threshold according to the learning state of the model during the learning process, thereby improving the utilization rate of the model using unlabeled samples; based on the dynamic confidence threshold, iteratively selecting high-confidence unlabeled samples to enter the labeled data set until the unlabeled sample set is empty.
[0013] Further, in step S1, it specifically includes: assuming a sensor data set It consists of a tag set and an unlabeled set Composition, T is the length of the sensor sequence, C represents the number of channels, K is the number of activity categories, N L Represents the number of labeled data, N URepresents the number of unlabeled data; generally speaking, N L >>N U This means that the number of unlabeled samples is often much larger than the number of labeled samples.
[0014] The supervised branch learns features from the labeled data set, which leads to overfitting due to the lack of data diversity. In order to increase the diversity and quantity of data, a random scaling method is used to increase the limited labeled data and improve the performance of the model; the deep model of the supervised branch includes a spatial feature encoding component, a temporal feature encoding component and a top classifier;
[0015] CNN is usually used to learn spatial features from sensor data. Although the feature maps learned by CNN usually contain redundant information, they usually lead to an increase in the number of parameters and computational complexity. To solve this problem, the present invention adopts partial convolution PConv to replace DWConv to reduce the high memory bandwidth requirement of DWConv; assuming that the input and output channels are equal, PConv only applies traditional convolution to r of the input channels for feature extraction, and the remaining 1-r channels remain unchanged; therefore, the FLOPs of PConv is h*w*k 2 *(rc) 2 , its memory bandwidth is calculated as
[0016] h*w*2(rc)+k 2 *(rc) 2 ≈h*w*2(rc),
[0017] Where r is the local partition ratio, and h*w*2(rc) represents the memory access amount of I / O operations; therefore, the computational cost of PConv is only r times that of traditional convolution. 2 , while also reducing the memory bandwidth to r times that of traditional convolution. Combining PConv with PWConv effectively exploits the inter-channel correlation and presents a T-shaped receptive field in the input feature map. This combination makes full use of filter redundancy, thereby saving computation. For feature maps stored continuously or in patterns, the first or last channel can be selected to represent the entire feature map.
[0018] Specifically, the spatial feature encoding component includes four main modules, each of which contains one or more basic blocks, and each basic module consists of one PConv and two PWConvs; batch normalization (BN) and GELU activation layers are embedded between the two PWConv layers to enhance feature representation and reduce information transmission delay; such a design strategy not only speeds up data processing, but also maintains feature diversity, greatly improving the performance and efficiency of the network in processing complex data.
[0019] Furthermore, in step S1, in order to learn temporal features from sensor data, the recent bidirectional GRU (Bi-GRU) is used as the key component after the spatial feature encoding component; compared with the popular long short-term memory (LSTM), GRU simplifies the model structure by merging the forget gate and the input gate, thereby reducing the parameter requirements during training. This not only reduces memory consumption but also improves execution efficiency. Bi-GRU is based on two key gating mechanisms: update gate and reset gate; the update gate is related to the input x t Combined, it determines the degree of retention of past hidden state information; the description of the forward update gate and the backward update gate is as follows:
[0020] z t =σ(W z x t +U z h t-1 ),
[0021] z' t =σ(W' z x t +U' z h t+1 ),
[0022] where z t and z' t represent the forward and backward update gates of the current time step t respectively; W, U, W' and U' are the weight matrices of the forward and backward gates; h t-1 and h t+1 are the forward hidden state and the backward hidden state; the sigmoid function σ is used to ensure that the output value of the gate is limited between 0 and 1;
[0023] Forward reset gate r t and the backward reset gate r' t According to the input x t and the linear combination of the corresponding hidden state, which determines how much previous state information to discard; mathematically, r t and r' t The description is as follows:
[0024] r t =σ(W r x t +U r h t-1 ),
[0025] r' t =σ(W' r x t +U' r h t+1 ),
[0026] Specifically, Bi-GRU relies on resetting the gate and the past hidden state to update its forward and backward candidate hidden states at each time step. and
[0027]
[0028]
[0029] Among them, tanh is an activation function used to convert input data into output between -1 and +1;
[0030] Then, at time step t, the Bi-GRU adjusts the forward and backward hidden states h by t and h' t :
[0031]
[0032]
[0033] In this way, the used Bi-GRU fully models the temporal dependencies of the activity data through forward and backward processing.
[0034] Further, in step S3, in order to identify high-confidence samples and integrate them into the training data set, traditional pseudo-labeling methods usually set a higher sample selection threshold. However, this method may limit the model's generalization ability and the ability to fully utilize useful information in unlabeled data. In the present invention, a dynamic confidence adjustment mechanism is adopted, which can dynamically adjust the threshold according to the development status of the model during the learning process, thereby improving the efficiency of the model in utilizing unlabeled samples. According to the dynamic confidence threshold, high-confidence unlabeled samples are iteratively selected to enter the labeled data set; the dynamic confidence aims to improve the pseudo-label generation process by coordinating global confidence and local confidence; the global threshold has two functions: one is to reflect the comprehensive learning state of the model based on the model's confidence in the unlabeled data; the second is to ensure that the global confidence is steadily improved and is not affected by the distortion caused by abnormal pseudo-labels; to this end, the global threshold is adjusted. It is set to the average confidence of all unlabeled data and then updated using the exponential moving average (EMA). The process is described as:
[0035]
[0036] Among them, p(y|x i ; θ) represents the predicted probability distribution of the model for the unlabeled samples after weak enhancement, represents the global confidence at the previous moment, λ is the decay coefficient of EMA, ranging from 0 to 1, and B represents the number of labeled and unlabeled samples in the batch; the local threshold is used to effectively grasp the diversity within the category and the similarity between categories, thereby mapping the learning state of the model for a specific category. The local threshold is given by It is set based on the model's confidence that the unlabeled data is classified into a specific category; then, the local threshold is dynamically updated using EMA, which reflects the model's ability to continuously evolve, and is written as:
[0037]
[0038] in, represents a set of probabilities associated with K categories, Represents the local confidence at the previous moment;
[0039] In order to eliminate cognitive biases within the model, the global confidence is fine-tuned using local thresholds. When the model performs below expectations in a specific category, the dynamic confidence mechanism automatically lowers the threshold; this method allows more samples to be added to the category, thereby promoting a more balanced learning process. As the model gradually improves its recognition ability for a certain category through continuous training, the dynamic confidence will also increase appropriately. This adjustment improves the accuracy of the model in selecting high-quality unlabeled samples. The dynamic confidence threshold τ is crucial for filtering pseudo-labels, and its calculation formula is as follows:
[0040]
[0041] Furthermore, the loss function of this method consists of three different parts: supervision loss L s , the consistency regularization loss L of the semi-supervised branch u and class imbalance loss L i ; The total loss function of the model Written
[0042]
[0043] Where β and λ are loss weights (β is set to 1 and λ is set to 0.05 by default);
[0044] The supervised loss measures the difference between the predicted output of the model and the true activity labels provided in the training data. In this paper, the standard cross entropy loss is used as the measure of supervised loss and is expressed as:
[0045]
[0046] The consistency regularization loss is calculated based on the difference between the model's predictions for different augmented views of the same input. This loss encourages the model to generate consistent predictions in weak and strong augmentation and penalizes the model for generating inconsistent predictions. Mathematically, the consistency regularization loss is The writing method is as follows
[0047]
[0048] Among them, q i and Q i represent the predicted probabilities of weakly enhanced (i.e., scaling) and strongly enhanced (i.e., amplitude expansion, time expansion, and scaling) samples, respectively, refers to the cross entropy loss, τ is the dynamic confidence;
[0049] Considering that the generated pseudo labels may cause class imbalance, the focal loss method is used to reduce the relative weight of easy-to-classify samples, thereby prompting the model to pay more attention to those challenging classification samples. Specifically, by introducing a modulation factor (1-p t ) γ To reduce the loss contribution of easy-to-classify samples and correspondingly increase the loss contribution of hard-to-classify samples; in addition, in order to further fine-tune the effectiveness of the focal loss, a balancing variable α is introduced t , which is used to balance the weights of positive and negative samples in the loss function, thereby effectively solving the class imbalance problem; class imbalance loss Mathematically written as:
[0050]
[0051] Among them, α t is the balance coefficient, which is used to adjust the weights of positive and negative samples in the loss function; γ is the adjustment coefficient, which is used to control the loss contribution of positive and negative samples; for parameter α t and γ, using the default values in Focal loss; p t is the model's predicted probability for the true label, described as follows:
[0052]
[0053] The present invention also provides a semi-supervised deep learning human activity recognition system.
[0054] The beneficial effects of the present invention are:
[0055] The semi-supervised deep learning human activity recognition method and system of the present invention adopts an adaptive confidence adjustment strategy, which can adjust the confidence value according to the current learning state of the model. In this way, unlabeled data can be better utilized without sacrificing accuracy, thereby significantly improving the accuracy of HAR.
[0056] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0058] Figure 1 It is a schematic diagram of the overall structure of the present invention;
[0059] Figure 2 It is the PConv convolution in the present invention;
[0060] Figure 3 It is the basic block of the deep model of the present invention;
[0061] Figure 4 The Bi-GRU in the deep model of the present invention;
[0062] Figure 5 t-SNE visualization of different semi-supervised methods (1% labeled data);
[0063] Figure 6 t-SNE visualization of the final layer features of ACS-HAR on the UCI-HAR dataset;
[0064] Figure 7 The pseudo-label distribution of each category when using 1% annotated data on the WISDM dataset;
[0065] Figure 8 The pseudo-label distribution of each class when using 2% annotated data on the WISDM dataset. DETAILED DESCRIPTION
[0066] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.
[0067] like Figure 1-Figure 4As shown, the present invention provides a method and system for human activity recognition based on semi-supervised deep learning. In the case of limited label data, the confidence threshold is dynamically adjusted according to the learning state of the model, and the unlabeled data is better utilized to significantly improve the accuracy of HAR. The technical solution of the present invention has three advantages: 1) It can effectively utilize unlabeled data. On the UCI-HAR dataset, the method only uses 1% of the labeled training data to achieve a HAR accuracy of up to 91.1%. 2) An adaptive confidence adjustment strategy is proposed, which can adjust the confidence value according to the current learning state of the model. In this way, it can make better use of unlabeled data without sacrificing accuracy. In this way, the proposed method can greatly reduce the amount of labeled data required. 3) A loss function is designed, which combines supervision loss as well as consistency regularization and class imbalance loss. Such a design can not only encourage the model to produce consistent predictions in weak and strong enhancements, punish the model for inconsistent predictions, but also help alleviate the class imbalance problem in pseudo-labeling. This is very helpful to improve the performance of the model, which is also proved in the ablation experiment.
[0068] Figure 1 The figure is a schematic diagram of the overall structure of the present invention. The present invention provides a semi-supervised deep learning method for human activity recognition, which includes the following steps: step S1: supervised training of small tag sensor signals; step S2: using a well-trained model to generate pseudo labels for unlabeled data; step S3: iteratively selecting pseudo label samples with high confidence and incorporating them into the training set until all high-quality samples are used up;
[0069] like Figure 1 As shown in the figure. This scheme takes sensor data from the built-in inertial sensors of smart devices (including accelerometers, gyroscopes, and magnetometers) as input. ACS-HAR first uses a small labeled dataset for supervised learning, and then uses the trained model to generate pseudo labels for unlabeled data. Subsequently, the system repeatedly selects pseudo-label samples with high prediction confidence and includes them in the training set until all high-quality samples are used.
[0070] ACS-HAR consists of two main branches: supervised branch and semi-supervised branch. Figure 1 The deep model in the supervision branch consists of a spatial feature encoder, a temporal feature encoder, and a classifier. This structure enables the model to deeply learn useful features of sensor data from both spatial and temporal dimensions. These features are then mapped to specific behavior categories by the top classifier. Figure 1As shown at the bottom, the semi-supervised branch adaptively adjusts the confidence threshold according to the learning state of the training model, thereby selectively incorporating high-confidence unlabeled samples into the training set. In this branch, the unsupervised loss of unlabeled samples is calculated by comparing the prediction differences under weak data augmentation and strong data augmentation.
[0071] Assume a sensor dataset It consists of a tag set and an unlabeled set T is the length of the sensor sequence, C is the number of channels, K is the number of activity categories, and N L Represents the number of labeled data, N U Represents the number of unlabeled data. Generally speaking, N L >>N U This means that the number of unlabeled samples is often much larger than the number of labeled samples.
[0072] In order to learn spatial features from sensor data, CNN is usually used. Although the feature maps learned by CNN usually contain redundant information, they usually lead to an increase in the number of parameters and computational complexity. To solve this problem, the partial convolution (PConv) introduced in this paper replaces DWConv to reduce the high memory bandwidth requirements of DWConv. Assuming that the input and output channels are equal, PConv only applies traditional convolution to r of the input channels for feature extraction, and the remaining 1-r channels remain unchanged. Therefore, the FLOPs of PConv is h*w*k 2 *(rc) 2 , its memory bandwidth can be calculated as h*w*2(rc)+k 2 *(rc) 2 ≈h*w*2(rc),
[0073] Where r is the local partition ratio, and h*w*2(rc) represents the memory access amount of I / O operations. Therefore, the computational cost of PConv is only r times that of traditional convolution. 2 , while also reducing the memory bandwidth to r times that of traditional convolution. Combining PConv with PWConv effectively exploits the inter-channel correlation and presents a T-shaped receptive field in the input feature map. This combination makes full use of filter redundancy, thereby saving computation. For feature maps stored continuously or in patterns, the first or last channel can be selected to represent the entire feature map.
[0074] Specifically, the spatial feature encoding component of the present invention includes four main modules, each of which may contain one or more fasterNet blocks, as shown in the figure. Each fasterNet module consists of one PConv and two PWConv. It is worth noting that batch normalization (BN) and GELU activation layers are embedded between the two PWConv layers to enhance feature representation and reduce information transmission delay. Such a design strategy not only speeds up data processing, but also maintains feature diversity, greatly improving the performance and efficiency of the network in processing complex data.
[0075] In order to learn temporal features from sensor data, this paper adopts the recent bidirectional GRU (Bi-GRU) as a key component after the spatial feature encoding component. Compared with the popular long short-term memory (LSTM), GRU simplifies the model structure by incorporating the forget gate and the input gate, thereby reducing the parameter requirements during training. This not only reduces memory consumption but also improves execution efficiency. Bi-GRU is based on two key gating mechanisms: update gate and reset gate. The update gate is related to the input x t Combined, it determines the degree of retention of past hidden state information. The description of the forward update gate and the backward update gate is as follows:
[0076] z t =σ(W z x t +U z h t-1 ),
[0077] z' t =σ(W' z x t +U' z h t+1 ),
[0078] where z t and z' t represent the forward and backward update gates for the current time step t. W, U, W' and U' are the weight matrices for the forward and backward gates. t-1 and h t+1 are the forward hidden state and the backward hidden state. The sigmoid function σ is used to ensure that the output value of the gate is limited between 0 and 1.
[0079] Forward reset gate r t and the backward reset gate r' t According to the input x t The linear combination of the corresponding hidden state determines how much previous state information to discard. Mathematically, r t and r' t The description is as follows:
[0080] rt =σ(W r x t +U r h t-1 ),
[0081] r' t =σ(W' r x t +U' r h t+1 ),
[0082] Specifically, Bi-GRU relies on resetting the gate and the past hidden state to update its forward and backward candidate hidden states at each time step. and
[0083]
[0084]
[0085] Among them, tanh is an activation function used to transform the input data into an output between -1 and +1.
[0086] Then, at time step t, the Bi-GRU adjusts the forward and backward hidden states h by t and h' t :
[0087]
[0088]
[0089] In this way, the used Bi-GRU fully models the temporal dependencies of the activity data through forward and backward processing.
[0090] In the present invention, a dynamic confidence adjustment mechanism is proposed, which can dynamically adjust the threshold according to the development status of the model during the learning process, thereby improving the efficiency of the model in utilizing unlabeled samples. According to the dynamic confidence threshold, high-confidence unlabeled samples are iteratively selected to enter the labeled data set. Dynamic confidence aims to improve the pseudo-label generation process by coordinating global confidence and local confidence. The global threshold has two functions: one is to reflect the comprehensive learning state of the model based on the model's confidence in the unlabeled data; the other is to ensure that the global confidence is steadily improved and is not affected by the distortion caused by abnormal pseudo-labels. To this end, the global threshold is set. It is set to the average confidence of all unlabeled data and then updated using the exponential moving average (EMA). This process can be described as:
[0091]
[0092] Among them, p(y|x i ; θ) represents the predicted probability distribution of the model for unlabeled samples after weak enhancement. represents the global confidence at the previous moment, λ is the decay coefficient of EMA, ranging from 0 to 1, and B represents the number of labeled and unlabeled samples in the batch.
[0093] The local threshold is used to effectively grasp the diversity within the category and the similarity between categories, thereby mapping the learning state of the model for a specific category. It is set based on the model's confidence that the unlabeled data is classified into a specific category. Then, the local threshold can be dynamically updated using EMA, which reflects the model's ability to continuously evolve. It is written as:
[0094]
[0095] in, represents a set of probabilities associated with K categories, Represents the local confidence at the previous moment.
[0096] In order to eliminate cognitive biases within the model, local thresholds are used to fine-tune the global confidence. When the model performs below expectations in a specific category, the dynamic confidence mechanism automatically lowers the threshold. This approach allows more samples to be added to the category, thereby promoting a more balanced learning process. As the model gradually improves its recognition ability for a certain category through continuous training, the dynamic confidence will also increase appropriately. This adjustment improves the accuracy of the model in selecting high-quality unlabeled samples. The dynamic confidence threshold τ is crucial for filtering pseudo-labels, and its calculation formula is as follows:
[0097]
[0098] The loss function of the method of the present invention contains three different parts: supervision loss L s , the consistency regularization loss L of the semi-supervised branch u and class imbalance loss L i The total loss function of the model is Can be written as
[0099]
[0100] Where β and λ are loss weights (by default, β is set to 1 and λ is set to 0.05).
[0101] The supervised loss measures the difference between the predicted output of the model and the true activity labels provided in the training data. In the invention, the standard cross entropy loss is used as the measure of supervised loss and is expressed as:
[0102]
[0103] The consistency regularization loss is calculated based on the difference between the model's predictions for different augmented views of the same input. This loss encourages the model to generate consistent predictions in weak and strong augmentations, and penalizes the model for generating inconsistent predictions. Mathematically, the consistency regularization loss is The writing method is as follows
[0104]
[0105] Among them, q i and Q i represent the predicted probabilities of weakly enhanced (i.e., scaling) and strongly enhanced (i.e., amplitude expansion, time expansion, and scaling) samples, respectively. refers to the cross entropy loss and τ is the dynamic confidence.
[0106] Considering that the generated pseudo labels may cause class imbalance, the focal loss method is used to reduce the relative weight of easy-to-classify samples, thereby prompting the model to pay more attention to those challenging classification samples. Specifically, this is achieved by introducing a modulation factor (1-p t ) γ To reduce the loss contribution of easy-to-classify samples and correspondingly increase the loss contribution of hard-to-classify samples. In addition, to further fine-tune the effectiveness of the focal loss, a balancing variable α is introduced t , which is used to balance the weights of positive and negative samples in the loss function, thereby effectively solving the class imbalance problem. Class imbalance loss Mathematically it can be written as:
[0107]
[0108] Among them, α t is the balance coefficient, which is used to adjust the weight of positive and negative samples in the loss function. γ is the adjustment coefficient, which is used to control the loss contribution of positive and negative samples. t and γ, we use the default values in Focal loss. t is the model's predicted probability for the true label, described as follows:
[0109]
[0110] The present invention is a semi-supervised deep learning method for human activity recognition, which aims to solve the problem that existing human activity recognition models require a large amount of labeled data to train the model. Collecting a large amount of labeled activity data usually requires a lot of manpower and is even costly. In this embodiment, the method of the present invention is evaluated with other state-of-the-art (SOTA) HAR methods using four public HAR datasets (including UCI-HAR, WISDM, mHealth, and PAMAP2). Among them, the baseline methods include four popular supervised learning baseline methods (i.e., CNN, DeepConvLSTM, LSTM-CNN, and CNN-GRU) and four SOTA semi-supervised learning methods (i.e., Autoencoder, Relation Prediction, Π-model, and Pseudo-Labeling).
[0111] This embodiment implements the proposed method in PyTorch. The number of training iterations is set to 120 and the batch size is 32. The AdamW optimizer is used to optimize the model with a learning rate of 0.0001 and a weight decay of 0.0003. During the training process, data enhancement techniques such as scaling, amplitude warping, and time warping are used. The F1 score is selected as the accuracy metric, and the model is trained five times to obtain the average value of the F1 score. In order to compare the performance of the method of the present invention with various baseline methods in different proportions of labeled data, this embodiment randomly selects a certain proportion of labeled samples to form a small labeled set, and discards the labels of the remaining data to form an unlabeled set. On each data set, different proportions of labeled samples (1\%, 2\%, 5\%, 10\%, 20\%, and 50\%) are used for training.
[0112] Table 1 Table 1 Comparison results of the percentage scores of ACS-HAR and SOTA methods on four datasets (bold numbers indicate the best performance)
[0113]
[0114] Table 2 Effects of different part ratios r and activation functions on ACS-HAR on UCI-HAR dataset
[0115]
[0116] Table 3. ACS-HAR performance using different loss function components on the UCI-HAR dataset
[0117]
[0118] Table 4 Effect of different β and γ values on ACS-HAR performance (F1 score)
[0119]
[0120] In this embodiment, the method proposed in the present invention is compared with several supervised techniques. The results are shown in Table 1. It can be seen from Table 1 that the method of the present invention (ACS-HAR) is significantly better than other supervised methods in all data sets and different proportions of labeled data. ACS-HAR achieved an accuracy of 87.4% on the UCI-HAR data set using 1% of labeled training data, which is much higher than DeepConvLSTM (82.9%), LSTM-CNN (73.54%), CNN-GRU (69.9%) and CNN (58.08%). When the proportion of labeled training data used increased from 1% to 50%, the performance of ACS-HAR increased from 87.4% to 97.1%. Other baseline methods also have similar trends. It is worth noting that due to the overfitting problem, using 100\% of labeled training data does not necessarily lead to the best performance. For example, the best performance of DeepConvLSTM is achieved by using 50%, not 100% of labeled data. On the WISDM, PAMAP2, and mHealth datasets, ACS-HAR also achieved the best accuracy when using different proportions of labeled training data. Among the baseline methods, DeepConvLSTM generally performs best on the WISDM and PAMAP2 datasets, but CNN-GRU outperforms DeepConvLSTM on the mHealth dataset.
[0121] In general, the proposed method can achieve the highest accuracy in almost all cases. It can achieve a relatively high accuracy (higher than 80%), while the experimental results are only 1%.
[0122] The method proposed in the present invention is compared with several popular semi-supervised methods in the embodiment, including autoencoder, relation prediction, Π-model and pseudo-labeling. Table 1 shows that the present method is superior to all semi-supervised baseline methods to a large extent. This is very obvious on the WISDM dataset, where ACS-HAR is 3% higher than pseudo-labeling and Π-model, and 16% higher than relation prediction and autoencoder. On other datasets, ACS-HAR also performs much better than Π-model, pseudo-labeling, relation prediction and autoencoder in most cases. In the baseline method, the performance of Π-model and pseudo-labeling is close, significantly better than autoencoder and relation prediction. This is because Π-model and pseudo-labeling use the same network as our method, and the latter is more powerful than the network used in autoencoder and relation prediction. Compared with supervised methods, when the backbone network used is the same, the performance of semi-supervised methods is generally better than supervised methods. This can be clearly seen by the fact that ACS-HAR with semi-supervised learning outperforms supervised methods by 3.5% on UCI-HAR, 3.6% on WISDM, 0.8% on PAMAP2, and 5.2% on mHealth. Pseudo-labeling and Π-model, which share the same backbone network as ACS-HAR, also outperform ACS-HAR without using unlabeled data. Similarly, the proposed method achieves very competitive accuracy using only 5\% of labeled data, which outperforms other semi-supervised baseline methods. When using 20% of labeled data, its performance is similar to that of supervised methods. This means that ACS-HAR has a very strong ability to learn from unlabeled data.
[0123] In order to qualitatively evaluate the feature embedding capability of the present invention, this embodiment uses t-SNE to visualize the learned feature embedding on the UCI-HAR dataset, as shown in Figure 5As shown. It should be noted that all models are trained on the same 1\% labeled training data in the UCI-HAR dataset, while the visualization features are obtained on some test data. All learned features are projected into two-dimensional space using t-SNE. It can be found that the method ACS-HAR of the present invention can learn more semantically separable features than other methods. The baseline method (ACS-HAR without using unlabeled data) performs poorly in distinguishing different clusters because the features of different categories are mixed (as shown in the black dotted ellipse). Although the Π-model and pseudo-labeling methods have improved in reducing misclassification by integrating unlabeled data, the magnitude is not large. The features in the pseudo-labeled and Π-model dotted ellipses are also difficult to separate. In contrast, the method proposed in the present invention greatly reduces the number of misclassified samples and effectively extracts distinctive features. This shows that ACS-HAR is a good general feature extractor for activity data.
[0124] In this example, in order to understand how the model in this method continues to improve as the training progresses, the features of the last layer of the model at different training durations are visualized. Figure 6 As shown in Figure 1, we can see that in the early stages of training, the feature distribution is relatively scattered. As training progresses, more high-confidence unlabeled samples are added to the training dataset, and the model's ability to distinguish different categories gradually increases. This is evidenced by the fact that samples with the same color are closer and samples with different colors are farther away. This shows that the proposed method can gradually improve its ability as more high-confidence unlabeled data samples are used.
[0125] This example analyzes the impact of different components in the total loss function on the ACS-HAR accuracy of this method, including consistency regularization loss, class imbalance loss, etc. Table 2 lists the results of using different loss function components. It can be seen that the consistency regularization loss L u and class imbalance loss i have a positive impact on the performance of ACS-HAR. In different proportions of labeled data, integrating these two losses can achieve the highest F1 score, and the improvement can reach 11.5% when less labeled data is available. The consistency regularization loss contributes more to the improvement in accuracy than the class imbalance loss, especially when less labeled data is used. However, as the amount of labeled data increases (such as 20% and 50%), the improvement in accuracy depends more on the labeled data itself, and the advantage of combining different loss terms is not obvious. This shows that the method proposed in the present invention is particularly suitable for situations where labeled data is scarce or costly to obtain.
[0126] This embodiment studies the effect of the partial ratio r of the partial convolution operation and the selection of the activation function (Act) on the performance of ACS-HAR. If the value of r is equal to 1, PConv will degenerate into a regular convolution, thereby enriching the understanding of the data, but consuming more computing resources. If the value of r is too small, although the consumption of computing resources can be reduced, the partial convolution cannot effectively capture spatial features, which may cause the performance of the model to deteriorate. In this embodiment, the partial ratio range considered is 1 to 1 / 8.
[0127] As shown in Table 3, in most cases, the performance of the ACS-HAR of the present invention is better when the partial ratio is set to 1 / 4. Therefore, the default setting r is 1 / 4, which can not only reduce the consumption of computing resources, but also effectively capture useful spatial features. In terms of activation function, the experimental results show that the GELU function is better than the ReLU. This may be because it provides higher nonlinearity, thus witnessing better performance.
[0128] This example analyzes the impact of the weight parameters β and γ in the total loss on the performance of ACS-HAR. Table 4 shows the results using different β and γ values, indicating that both β and γ have an impact on the performance of ACS-HAR, and this impact is more serious when less labeled data is used. In most cases, ACS-HAR performs best when the β value is set to 1.0. At the same time, when the labeled data is insufficient, γ performs well in the range of [0.05, 0.5]. Therefore, β and γ are set to 1.0 and 0.05 by default, which means that when the labeled data is insufficient, the model will pay more attention to unlabeled data and class imbalance problems.
[0129] In order to verify the effectiveness of the method of the present invention in solving the problem of pseudo-label category imbalance, this embodiment performs an ablation experiment on the WISDM dataset. It should be noted that the UCI-HAR dataset used by default for ablation research does not have a category imbalance problem because the number of samples in each category is almost equal. Figure 7 and Figure 8 The pseudo-label distribution of each category is shown when using 1% and 2% of the labeled data on the WISDM dataset. Compared with other methods, the method of the present invention can generate more pseudo-labels for categories with fewer samples and fewer pseudo-labels for the head category. However, the two popular semi-supervised learning methods, pseudo-label and Π model, perform poorly in generating pseudo-labels for categories with fewer samples, especially in the upstairs, downstairs, sitting and standing categories. This means that ACS-HAR can solve the class imbalance problem well and reduce the bias of the model.
[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A semi-supervised deep learning method for human activity recognition, characterized in that: The method comprises the following steps: S1, supervised training of small tag sensor signals in smart devices; S2. Generate pseudo labels for unlabeled data using the trained model; S3, iteratively select pseudo-label samples with high confidence and include them in the training set until all high-quality samples are used; S4. Use the trained model to identify human activities.
2. The method for human activity recognition based on semi-supervised deep learning according to claim 1, characterized in that: In step S1, during the supervised training phase of the small tag sensor signal, the processed dataset From the inertial sensors built into smart devices, including accelerometers, gyroscopes, and magnetometers; During supervised training, sensor signals is input into the supervision branch for feature encoding; the supervision branch consists of a spatial feature encoder, a temporal feature encoder, and a classifier; then, these learned features are mapped to specific behavior categories through the top classifier, where and denote the labeled sample set and the unlabeled sample set respectively, Represents the sensor signal, Indicates the related activity category.
3. The method for human activity recognition based on semi-supervised deep learning according to claim 2, characterized in that: In step S2, the trained model is used to generate pseudo labels for the unlabeled data, including implementing two different enhancement strategies for the unlabeled samples: weak enhancement and strong enhancement. It is then input into the trained supervision branch to generate pseudo labels.
4. The method for human activity recognition based on semi-supervised deep learning according to claim 3, characterized in that: In step S3, the iterative selection of high-confidence pseudo-label samples and inclusion in the training set until all high-quality samples are used up includes: using an adaptive confidence adjustment mechanism to dynamically adjust the threshold according to the learning state of the model during the learning process, thereby improving the utilization rate of the model using unlabeled samples; based on the dynamic confidence threshold, iteratively selecting high-confidence unlabeled samples to enter the labeled data set until the unlabeled sample set is empty.
5. The method for human activity recognition based on semi-supervised deep learning according to claim 2, characterized in that: In step S1, specifically including: assuming a sensor data set It consists of a tag set and an unlabeled set Composition, T is the length of the sensor sequence, C represents the number of channels, K is the number of activity categories, N L Represents the number of labeled data, N U represents the amount of unlabeled data; The supervision branch learns features from the labeled data set and uses random scaling to increase the limited labeled data to improve the performance of the model. The deep model of the supervision branch includes a spatial feature encoding component, a temporal feature encoding component, and a top classifier. PConv is used to replace DWConv to reduce the high memory bandwidth requirement of DWConv. Assuming that the input and output channels are equal, PConv only applies traditional convolution to r of the input channels for feature extraction, and the remaining 1-r channels remain unchanged. Therefore, the FLOPs of PConv is h*w*k 2 *(rc) 2 , its memory bandwidth is calculated as h*w*2(rc)+k 2 *(rc) 2 ≈h*w*2(rc), Where r is the local partition ratio, h*w*2(rc) represents the memory access amount of I / O operation; Specifically, the spatial feature encoding component includes four main modules, each of which contains one or more basic blocks, and each basic module consists of one PConv and two PWConvs; batch normalization and GELU activation layers are embedded between the two PWConv layers to enhance feature representation and reduce information transmission delay.
6. The method for human activity recognition based on semi-supervised deep learning according to claim 5, characterized in that: In step S1, in order to learn temporal features from sensor data, a bidirectional GRU is used as a key component after the spatial feature encoding component; Bi-GRU is based on two key gating mechanisms: update gate and reset gate; the update gate is related to the input x t Combined, it determines the degree of retention of past hidden state information; the description of the forward update gate and the backward update gate is as follows: z t =σ(W z x t +U z h t-1 ), z' t =σ(W' z x t +U' z h t+1 ), where z t and z' t represent the forward and backward update gates of the current time step t respectively; W, U, W' and U' are the weight matrices of the forward and backward gates; h t-1 and h t+1 are the forward hidden state and the backward hidden state; the sigmoid function σ is used to ensure that the output value of the gate is limited between 0 and 1; Forward reset gate r t and the backward reset gate r' t According to the input x t The linear combination of the corresponding hidden state determines how much previous state information to discard; r t and r' t The description is as follows: r t =σ(W r x t +U r h t-1 ), r' t =σ(W' r x t +U' r h t+1 ), Specifically, Bi-GRU relies on resetting the gate and the past hidden state to update its forward and backward candidate hidden states at each time step. and Among them, tanh is an activation function used to convert input data into output between -1 and +1; Then, at time step t, the Bi-GRU adjusts the forward and backward hidden states h by t and h' t : The used Bi-GRU fully models the temporal dependencies of activity data through forward and backward processing.
7. The method for human activity recognition based on semi-supervised deep learning according to claim 6, characterized in that: In step S3, in order to identify high-confidence samples and integrate them into the training dataset, a dynamic confidence adjustment mechanism is adopted. According to the dynamic confidence threshold, high-confidence unlabeled samples are iteratively selected to enter the labeled dataset; the dynamic confidence aims to improve the pseudo-label generation process by coordinating global confidence and local confidence; The global threshold has two functions: first, it reflects the overall learning state of the model based on the model's confidence in the unlabeled data; second, it ensures that the global confidence increases steadily and is not affected by distortion caused by abnormal pseudo-labels; To do this, set the global threshold It is set to the average confidence of all unlabeled data and then updated using exponential moving average. The process is described as: Among them, p(y|x i ; θ) represents the predicted probability distribution of the model for the unlabeled samples after weak enhancement, represents the global confidence at the previous moment, λ is the decay coefficient of EMA, ranging from 0 to 1, and B represents the number of labeled and unlabeled samples in the batch; the local threshold is given by It is set based on the model's confidence that the unlabeled data is classified into a specific category; then, the local threshold is dynamically updated using EMA, which reflects the model's ability to continuously evolve, and is written as: in, represents a set of probabilities associated with K categories, Represents the local confidence at the previous moment; The global confidence is fine-tuned using a local threshold. When the model performs worse than expected in a specific category, the dynamic confidence mechanism automatically lowers the threshold. The dynamic confidence threshold τ is crucial for filtering pseudo-labels, and its calculation formula is as follows:
8. The method for human activity recognition based on semi-supervised deep learning according to claim 7, characterized in that: The loss function of this method consists of three different parts: supervision loss L s , the consistency regularization loss L of the semi-supervised branch u and class imbalance loss L i ; The total loss function of the model Written Among them, β and λ are loss weights; The standard cross entropy loss is used as the metric for supervision loss and is expressed as: The consistency regularization loss is calculated based on the difference between the model’s predictions for different augmented views of the same input. This loss encourages the model to generate consistent predictions in weak and strong augmentations and penalizes the model for generating inconsistent predictions. The writing method is as follows Among them, q i and Q i Represent the predicted probabilities of weakly enhanced and strongly enhanced samples, respectively. refers to the cross entropy loss, τ is the dynamic confidence; The focal loss method is used to reduce the relative weight of easy-to-classify samples. Specifically, a modulation factor (1-p t ) γ To reduce the loss contribution of easy-to-classify samples and correspondingly increase the loss contribution of hard-to-classify samples; in addition, in order to further fine-tune the effectiveness of the focal loss, a balancing variable α is introduced t , which is used to balance the weights of positive and negative samples in the loss function, thereby effectively solving the class imbalance problem; class imbalance loss Mathematically written as: Among them, α t is the balance coefficient, which is used to adjust the weights of positive and negative samples in the loss function; γ is the adjustment coefficient, which is used to control the loss contribution of positive and negative samples; for parameter α t and γ, using the default values in Focal loss; p t is the model's predicted probability for the true label, described as follows:
9. A semi-supervised deep learning human activity recognition system, characterized by: The system adopts the method as claimed in any one of claims 1 to 8.
Citation Information
Patent Citations
Human body behavior recognition method based on consistent semi-supervised deep learning
CN115188022A
Semi-supervised video fire detection method based on consistency regularization and distribution alignment
CN117274881A
Multi-stage human body activity identification method based on small amount of label data
CN117932434A
Systems and methods for semi-supervised learning with contrastive graph regularization
US20220156591A1
Cited By
Seismic data augmentation method and system based on adversarial network and semi-supervised learning
CN120610309A
