Human activity recognition comprehensive optimization method based on multi-scale metric learning

Abstract

The application provides a human activity recognition comprehensive optimization method based on multi-scale metric learning, and relates to the technical field of human activity recognition of sensors. After data collected by a sensor is processed, over-sampling of boundary samples of a minority class in a data set is performed by using a Borderline SMOTE up-sampling algorithm, a new MAG-Res2Net neural network model is constructed to extract, fuse and optimize feature data, classification training of the features is performed by combining a loss function method, and finally, the network model is evaluated and improved to recognize human activities, so that the problems of data imbalance, difficulty in feature extraction and low classification precision in the human activity recognition process are solved.

Description

Technical Field

[0001] This invention belongs to the field of sensor-based human activity recognition technology, specifically relating to a comprehensive optimization method for human activity recognition based on multi-scale metric learning. Background Technology

[0002] Human activity recognition refers to the technology of recognizing and classifying human postures, movements and behaviors, aiming to realize personalized health management, intelligent security monitoring and other applications, thus playing an important role in people's daily life ([1] Zhang Yang, Yao Dengfeng, "A review of human behavior recognition and classification methods", Power System Protection and Control, vol.47, no.24, pp.1-11, Dec.2019. (DOI:10.26914 / c.cnkihy.2019.004425)). Thanks to the development of smart devices, convenient high-performance detection devices are gradually being used in people's daily life, such as smart homes, motion tracking and medical health systems for monitoring and preventing diseases in the elderly ([2] K.Guan, M.Shao, and S.Wu, "A Remote HealthMonitoring System for the Elderly Based on Smart Home Gateway", Journal of Healthcare Engineering, 2017. https: / / doi.org / 10.1155 / 2017 / 5843504.). Given the pressing issue of global population aging, and the challenge of preparing healthcare systems to cope with demographic changes worldwide, HAR has emerged as one of the most promising solutions.

[0003] There are two main technical methods for human activity recognition: video-based systems and sensor-based systems. Video-based systems use computer vision technology to analyze and recognize human dynamics; sensor-based systems measure and analyze human movement through data collected by sensors. Compared with video systems, sensor systems can more accurately obtain human dynamic information and position information ([3] Zhang Lin et al., "Pedestrian Dead Estimation and Positioning Method Based on Human Activity Recognition Assistance", Radio Engineering, vol.52, no.10, pp.1803-1812, Oct.2022. (DOI:10.13164 / re.2022.1803)). Currently, due to privacy issues, sensor-based human activity recognition systems have dominated the application of monitoring our daily activities. Sensors can be embedded in mobile phones, watches, furniture and walls, etc., and can collect motion information in people's daily lives ([4] A. Ferrari, D. Micucci, M. Mobilio, and P. Napoletano, "Trends in Human Activity Recognition Using Smartphones," Journal of Reliable Intelligent Environments, vol.7, no.3, Sep.2021, pp.189-213, doi:10.1007 / s40860-021-00147-0.). In the early days, traditional machine learning methods, such as decision tree (DT), support vector machine (SVM), random forest (RF) and Naive Bayes (NB), have made considerable progress in sensor-based human activity recognition. However, feature extraction in machine learning is highly dependent on manual work, so more advanced feature extraction methods are needed to improve recognition accuracy ([5] TTAlemayoh, JHLee, and S. Okamoto, "New Sensor Data Structuring for Deeper Feature Extraction in Human Activity Recognition," Sensors, vol.21, no.8, Apr.2021, pp.1-17, doi:10.3390 / s21082814.).

[0004] Unlike traditional machine learning, deep learning has powerful nonlinear modeling capabilities and can automatically extract features, which provides the possibility for automating activity recognition tasks ([6] Almaslukh, B.; Al Muhtadi, J.; Artoli, AMA robust convolutional neural network for online smartphone-based human activity recognition. J. Intell. Fuzzy Syst. 2018, 35, 1609–1620.). In recent years, many deep learning-based action recognition methods have been proposed. However, despite the good success that deep learning has achieved in the field of human activity recognition, experimental results still show that it still faces some technical challenges in the field of HAR ([7] J.Wang, Y.Chen, S.Hao, X.Peng, and L.Hu, "Deep Learning for Sensor-Based Activity Recognition: A Survey," Pattern Recognition Letters, Deep Learning for Pattern Recognition, vol.119, pp.3-11, Mar.2019. doi:10.1016 / j.patrec.2018.02.010.).

[0005] In recent years, with the development of neural network models, more and more patents have adopted deep learning to solve the aforementioned machine learning problems. For example, patent US20180078296A1, "Method and System for Classifying and Predicting Data using Deep Learning," uses deep convolutional neural networks for feature extraction and classification. It also proposes different deep learning models and training methods for different data types and application scenarios, effectively improving accuracy and training efficiency. However, this patent still has shortcomings in dealing with imbalanced data, and the training and optimization of the model are areas that need further improvement in its application. Another example is patent US20190057928A1, "Method and System for Emotion Classification of Speech Signals using Deep Neural Networks," which proposes a deep learning method combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs). By extracting features and modeling sequences in signals, it can effectively identify and classify different states. This method is superior to traditional methods in terms of accuracy and efficiency, and requires less data and computational resources in its implementation. However, this patent still has certain limitations when dealing with situations where there are large differences within the sample class. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a comprehensive optimization method for human activity recognition based on multi-scale metric learning, aiming to solve the issues of data imbalance, difficulty in feature extraction, and low classification accuracy in the human activity recognition process. The method includes the following steps:

[0007] Step 1: Collect signal data about human activities, organize the data into a table, and then handle missing values;

[0008] Step 1.1: Use accelerometers, gyroscopes and wireless sensors to record sensor data during human activities, including: acceleration, angular velocity, signal delay, signal strength information, and organize the data into a table.

[0009] Step 1.2: Missing human activity data is filled in using a linear interpolation algorithm;

[0010] Step 2: Extract the time series data from the sensor data and preprocess the time series data;

[0011] Step 2.1: Extract the time series data from the sensor data using the sliding window method;

[0012] Step 2.2: Preprocess the time series data using data cleaning, data standardization, data segmentation, data labeling, and data augmentation methods;

[0013] Step 3: When there is a class imbalance problem in the time series dataset after Step 2, the Borderline-SMOTE (Borderline Synthetic Minority Over-sampling Technique) upsampling algorithm is used to oversample the boundary samples of the minority class in the dataset to obtain a dataset with a balanced number of class samples.

[0014] Step 3.1: Based on different datasets and training tasks, the two branches of the Borderline-SMOTE upsampling algorithm, Borderline-SMOTE1 and Borderline-SMOTE2, are used to oversample the minority class samples.

[0015] Step 3.1.1: Divide minority class samples into 3 categories based on the proportion of minority class samples in neighboring samples: more than half are safe samples, less than half are dangerous samples, and if there are no minority class samples, they are judged as noise. Only oversample minority class samples that exhibit the behavior of dangerous samples.

[0016] Step 3.1.2: Calculate the dangerous minority class sample point a i The distance matrix D is obtained by equating the distances between the i-th sample point and all sample points in the minority class sample set S. D(i, p) represents the Euclidean distance between the i-th sample point and the p-th sample point. Each row of the distance matrix D is sorted, and the columns corresponding to the first k minimum values ​​are found and stored in a list. This list is the k nearest neighbors of the i-th sample point.

[0017] Step 3.1.3: When the minority class samples are spatially concentrated, use Borderline-SMOTEl to generate new samples for the danger points, and randomly select j minority class samples from the k nearest neighbors as the nearest neighbor samples. When the minority class samples are spatially dispersed, use Borderline-SMOTE2 to generate new samples for the danger points, and randomly select a sample from the k nearest neighbors as the nearest neighbor sample, without paying attention to the sample category.

[0018] Step 3.1.4: Calculate the risky minority class sample a i Each sample a of its chosen nearest neighbor ij The difference vector is calculated using the following formula:

[0019] d ij =a i -a ij(1)

[0020] In the formula, a i For the i-th dangerous minority class sample, a ij Let d be the j-th nearest neighbor sample corresponding to the i-th minority class sample. ij For the difference vector between minority class samples and nearest neighbor samples, when using Borderline-SMOTE2, j = l;

[0021] Step 3.1.5: Synthesize new minority class samples, using the following formula:

[0022] h ij =a i +d ij ×rand(0,R) ij (2)

[0023] In the formula, h ij For the new minority class sample, d ij R is the difference vector between minority class samples and nearest neighbor samples. ij It is a constant;

[0024] Step 3.2: Verify whether the number of samples belonging to each class defined in Step 3.1.1 is balanced in the new minority class samples: If balanced, merge the new minority class samples with the majority class samples in the original dataset to form a balanced dataset. If unbalanced, the Borderline-SMOTEl or Borderline-SMOTE2 branch algorithm in the Borderline SMOTE upsampling algorithm needs to be used again to generate samples in order to achieve the effect of balanced sample numbers and then form a balanced dataset.

[0025] Step 4: Construct a novel multi-scale attention mechanism gated residual deep convolutional neural network model MAG-Res2Net, and use the neural network model to extract features from the dataset;

[0026] Step 4.1: Based on the ResNet module, a new deep convolutional neural network model is constructed by sequentially adding multi-scale filters, gating modules, and ECA attention modules, including:

[0027] Step 4.1.1: Build a ResNet module by replacing the n-channel 3×3 convolution kernels with a set of filter banks smaller than 3×3 convolution kernels, each of which has w channels, and connect the filter banks in a hierarchical manner similar to residuals to form a Res2Net module;

[0028] Step 4.1.2: Add a gated module after each 3×3 convolution to form a Gated-Res2Net module;

[0029] Step 4.1.3: Combine the Gated-Res2Net module and the ECA attention module to form a novel multi-scale attention mechanism gated residual deep convolutional neural network model MAG-Res2Net;

[0030] Step 4.2: Feed the dataset processed in Step 3 into the established network model for feature extraction, feature combination, and prediction, including:

[0031] Step 4.2.1: The dataset is divided into i parts x after a 1×1 convolution. i The data other than the first data x1 is fed into a 3×3 convolution;

[0032] Step 4.2.2: After processing in Step 4.2.1, the gating module selects features with a threshold range of l ± δ from the data obtained. The selected effective features are then mapped and passed to the next layer of the network model. The feature mapping formula includes:

[0033] f2(x) = concat(a(X), a(y)) i-1 ), a(b i (3)

[0034] f1(x)=a(f2(x)) (4)

[0035] g i =tanh(f1(x)) (5)

[0036] In the formula, x is the input feature map, representing the current input data, X is the original feature map, and y i-1 Let b be the output feature map of the previous layer, representing the output result of the previous layer. i For the next input feature map, a() is a 3×3 convolutional layer, a(X) is the feature map obtained by passing the original feature map X through a 3×3 convolutional layer, and a(y) is the feature map obtained by passing the original feature map X through a 3×3 convolutional layer. i -1) represents the output feature map y of the previous layer. i-1 The feature map obtained after a 3×3 convolution layer, a(b) i ) represents the feature mapping b of the current input. i The feature map is obtained after a 3×3 convolution layer. `concat` is the vector concatenation function. `f2(x)` is the feature map obtained after concatenating the feature maps. `f1(x)` is the feature map obtained after a fully connected neural network layer. `tanh()` is the hyperbolic tangent function in the activation function. i The gated vector is obtained by performing a nonlinear transformation on f1(x);

[0037] Step 4.2.3: The data processed in step 4.2.2 is mapped to form i copies of data y. iy i After stacking, the data is fed into a 1×1 convolution, with the following mapping formula:

[0038]

[0039] In the formula, c i Let be the i-th input data, representing the input of the current layer, d i Let K be the i-th output feature map, representing the output result of the current layer. i For convolution operations, the input data is processed by convolution to obtain feature representations, g i Here, s is the gate vector, s is the number of layers in the neural network, and s represents the depth of the entire network.

[0040] Step 4.2.4: The data after 1×1 convolution is fed into the ECA attention module. The ECA attention module performs global average pooling on the input feature data to obtain a feature vector with l channels, and then performs one-dimensional convolution on the feature vector.

[0041] Step 4.2.5: Apply the Sigmoid activation function to the vector after one-dimensional convolution to obtain a vector with values ​​between 0 and 1;

[0042] Step 4.2.6: Multiply the obtained vector element-wise with the original input feature data to obtain the weighted feature data;

[0043] Step 5: Train the network model by combining the center loss and cross-entropy loss functions and the defined optimization algorithm, and input the collected data into the trained model to classify human activities;

[0044] Step 5.1: Add a center loss calculation layer to the network model, and use the center loss function and cross-entropy loss function to calculate the difference between sample features and intra-class centers;

[0045] The central loss function is as follows:

[0046]

[0047] In the formula, Loss cr The center loss function is used to measure the difference between sample features and class centers, where N is the number of samples and z is the center of the class. i Let be the feature vector of the i-th sample, representing the output of the network, and center be the class center, representing the center of the class to which the current sample belongs;

[0048] The cross-entropy loss function is as follows:

[0049]

[0050] In the formula, Loss ce The cross-entropy loss function measures the difference between the model output and the true label, where n is the number of samples and p(z) is the number of samples. i Let q(z) be the true label probability distribution of the i-th sample, representing the probability that the sample belongs to each category. i Let be the predicted label probability distribution of the i-th sample, representing the model's predicted probability that the sample belongs to each category;

[0051] Step 5.2: Define the optimization algorithm: When using the AdamW optimization algorithm to train the network model and find the optimal training batch, if the optimal training batch exceeds α, the Lion optimization algorithm is used. The optimal learning rate on the AdamW optimization algorithm is mapped to the learning rate of the Lion optimization algorithm in a ratio of β:1, and training is based on this learning rate. Otherwise, the AdamW optimization algorithm is used for training.

[0052] Step 5.3: Use K-fold cross-validation to divide the output data of the network model obtained in Step 4 into a training set and a validation set;

[0053] Step 5.4: Train the network model using the training set: Load the center loss function and cross-entropy loss into the optimization algorithm respectively. Use the cross-entropy loss function to optimize the classification accuracy of the model, and the center loss function to optimize the clustering effect of the feature representation.

[0054] Step 5.5: Calculate the total loss function after training each batch of models and update the weight parameters of the network model through backpropagation. The formula for calculating the total loss function is as follows:

[0055] Loss total =Loss ce +λLoss cr (9)

[0056] In the formula, λ is the center loss weight, and Loss total The summation of the loss functions is used as the final loss function in model training;

[0057] Step 5.6: Evaluate the trained network model using the validation set: Calculate the values ​​of the cross-entropy loss function and the center loss function of the trained network model. Based on the classification accuracy and the value of the center loss function on the validation set, determine whether the model's performance meets the requirements. If it meets the requirements, store it as the final network model. If it does not meet the requirements, adjust the model's hyperparameters and retrain the model.

[0058] Step 5.7: Input all the data collected by the sensors in step 1 into the final network model to identify human activities.

[0059] The beneficial effects of this invention are:

[0060] This invention provides a comprehensive optimization method for human activity recognition based on multi-scale metric learning. By recognizing human posture, movement and behavior through a multi-scale metric learning method, it solves the problems of data imbalance, difficulty in feature extraction and low classification accuracy in the process of human activity recognition.

[0061] First, the Borderline SMOTE upsampling algorithm is applied to the dataset based on the multi-scale metric learning method to oversample the data samples, effectively reducing various adverse phenomena caused by data imbalance, improving the accuracy and robustness of the deep learning model, and reducing the risk of misclassification.

[0062] Secondly, a novel neural network was constructed based on the multi-scale metric learning method, which solved the problem of deep feature extraction, improved the accuracy and robustness of the model, and is suitable for multiple application scenarios.

[0063] Finally, based on the multi-scale metric learning method, the loss function and optimization algorithm in deep learning are improved by combining the center loss and cross-entropy loss functions to reduce intra-class differences and increase inter-class differences, thereby improving the classification performance and robustness of the model, effectively alleviating the overfitting problem of some classes, and improving training efficiency and accuracy. Attached Figure Description

[0064] Figure 1 This is a flowchart of the Borderline SMOTE upsampling algorithm execution.

[0065] Figure 2 The flowchart for the execution of the Gated-Res2Net module.

[0066] Figure 3 This is a flowchart illustrating the process of combining the loss function with model training.

[0067] Figure 4 This is a flowchart of the overall optimization method for human activity recognition based on multi-scale metric learning.

[0068] Figure 5 This is a diagram of the ResNet architecture.

[0069] Figure 6 This is a diagram of the Res2Net architecture.

[0070] Figure 7 This is a diagram of the Gated-Res2Net architecture.

[0071] Figure 8 This is a diagram of the MAG-Res2Net architecture. Detailed Implementation

[0072] To make the advantages and technical solutions of the present invention clearer and more explicit, the present invention will be further described below in conjunction with the accompanying drawings and specific implementation examples.

[0073] This invention proposes a comprehensive optimization method for human activity recognition based on multi-scale metric learning, the overall flowchart of which is shown below. Figure 4 As shown, it includes the following steps:

[0074] Step 1: Collect signal data about human activities, organize the data into a table, and then handle missing values;

[0075] Step 1.1: Use accelerometers, gyroscopes and wireless sensors to record sensor data during human activities, including: acceleration, angular velocity, signal delay, signal strength information, and organize the data into a table.

[0076] Step 1.2: Missing human activity data (represented by NaN or 0) are filled with missing values ​​using a linear interpolation algorithm;

[0077] Step 2: Extract the time series data from the sensor data and preprocess the time series data;

[0078] Preprocessing makes features easier for the network model to process, thus improving the model's performance.

[0079] Step 2.1: Extract the time series data from the sensor data using the sliding window method;

[0080] Step 2.2: Preprocess the time series data using data cleaning, data standardization, data partitioning, and data augmentation methods;

[0081] A sliding window method was used for data segmentation. In the WISDM dataset, the window length was set based on a sampling rate of 20Hz, so the window length was 20. Segmentation was based on user ID; users with IDs less than or equal to 27 were used in the training set, and the rest were used in the test set. Different types of human activities were labeled during data annotation for subsequent model training and testing. In the UCI-HAR dataset, since the dataset had already undergone the above processing, no further preprocessing was performed.

[0082] Step 3: When class imbalance exists in the time series dataset processed in Step 2, the Borderline-SMOTE (Borderline Synthetic Minority Over-sampling Technique) upsampling algorithm is used to oversample the boundary samples of the minority class in the dataset, resulting in a dataset with balanced class sample numbers. The flowchart is as follows. Figure 1 As shown;

[0083] In the WISDM dataset, the preprocessed dataset class ratio is 16991:4972:1936:2400:13620:4005, and after processing with the Borderline SMOTE algorithm, it becomes 16991:16991:16991:16991:16991:16991:16991.

[0084] To prevent dataset contamination, this invention only upsamples the training set data samples without making any changes to the test set samples.

[0085] Step 3.1: Based on different datasets and training tasks, the two branches of the Borderline-SMOTE upsampling algorithm, Borderline-SMOTE 1 and Borderline-SMOTE 2, are used to oversample the minority class samples.

[0086] Step 3.1.1: Divide minority class samples into 3 categories based on the proportion of minority class samples in neighboring samples: more than half are safe samples, less than half are dangerous samples, and if there are no minority class samples, they are judged as noise. Only oversample minority class samples that exhibit the behavior of dangerous samples.

[0087] Step 3.1.2: Calculate the dangerous minority class sample point a i The distance matrix D is obtained by equating the distances between the i-th sample point and all sample points in the minority class sample set S. D(i, p) represents the Euclidean distance between the i-th sample point and the p-th sample point. Each row of the distance matrix D is sorted, and the columns corresponding to the first k minimum values ​​are found and stored in a list. This list is the k nearest neighbors of the i-th sample point.

[0088] Step 3.1.3: When the minority class samples are spatially concentrated (in this invention, the parameter value for determining the number of samples located at the boundary of the dataset is set to 5, and a value greater than 5 is defined as concentrated), Borderline-SMOTE1 is used to generate new samples for the danger points, and j minority class samples are randomly selected from the k nearest neighbors as the nearest neighbor samples. When the minority class samples are spatially dispersed (less than 5 is defined as dispersed), Borderline-SMOTE2 is used to generate new samples for the danger points, and a sample is randomly selected from the k nearest neighbors as the nearest neighbor sample, without regard to the sample category.

[0089] Step 3.1.4: Calculate the risky minority class sample a i Each sample a of its chosen nearest neighbor ij The difference vector is calculated using the following formula:

[0090] dij =a i -a ij (1)

[0091] In the formula, a i For the i-th dangerous minority class sample, a ij Let d be the j-th nearest neighbor sample corresponding to the i-th minority class sample. ij For the difference vector between minority class samples and nearest neighbor samples, when using Borderline-SMOTE2, j = l;

[0092] Step 3.1.5: Synthesize new minority class samples, using the following formula:

[0093] h ij =α i +d ij ×rand(0,R) ij (2)

[0094] In the formula, h ij For the new minority class sample, d ij R is the difference vector between minority class samples and nearest neighbor samples. ij It is a constant (the value in this invention is 0.5 or 1);

[0095] Step 3.2: Verify whether the number of samples belonging to each class defined in Step 3.1.1 is balanced in the new minority class samples: If balanced, merge the new minority class samples with the majority class samples in the original dataset to form a balanced dataset. If unbalanced, the Borderline-SMOTEl or Borderline-SMOTE2 branch algorithm in the Borderline SMOTE upsampling algorithm needs to be used again to generate samples in order to achieve the effect of balanced sample numbers and then form a balanced dataset.

[0096] Step 4: Construct a novel multi-scale attention mechanism gated residual deep convolutional neural network model MAG-Res2Net, and use the neural network model to extract features from the dataset;

[0097] A gated residual neural network model with a multi-scale attention mechanism is constructed. ECA (Electronic Assistive Computation) is used as the attention module at the end of each residual block, and gating is employed to filter important features. Multi-scale filters, gating mechanisms, and ECA attention modules are sequentially added to ResNet to construct a residual layer. A convolutional block consists of a residual layer and two consecutive 1×1 convolutions. The total number of layers depends on the dataset and time cost; for example, 50 layers perform better on the UCI-HAR dataset, while 42 layers perform better on the WISDM dataset. The number of layers here refers only to convolutional and fully connected layers; activation layers or pooling layers are not included.

[0098] Step 4.1: Based on the ResNet module, a new deep convolutional neural network model is constructed by sequentially adding multi-scale filters, gating modules, and ECA attention modules, including:

[0099] Step 4.1.1: Create a ResNet module, the structure diagram is as follows. Figure 5 As shown, a Res2Net module is formed by replacing the n-channel 3×3 convolution kernels with a set of filter banks smaller than the 3×3 kernels, each with w channels, and connecting the filter banks in a residual-like hierarchical manner. The structure diagram is shown below. Figure 6 As shown;

[0100] Step 4.1.2: Add a gated module after each 3×3 convolution to form a Gated-Res2Net module, as shown in the structure diagram below. Figure 7 As shown;

[0101] Step 4.1.3: Combine the Gated-Res2Net module and the ECA attention module to form a novel multi-scale attention mechanism gated residual deep convolutional neural network model, MAG-Res2Net, as shown in the structure diagram. Figure 8 As shown;

[0102] Step 4.2: Feed the dataset processed in Step 3 into the established network model for feature extraction, feature combination, and prediction. The flowchart is as follows: Figure 2 As shown, it includes:

[0103] Step 4.2.1: The dataset is divided into i parts x after a 1×1 convolution. i The data other than the first data x1 is fed into a 3×3 convolution;

[0104] Step 4.2.2: After processing in Step 4.2.1, the gating module selects features with a threshold range of 1 ± δ (δ is set to 0.5 in this invention), maps the selected effective features, and then passes them into the next layer of the network model. The feature mapping formula includes:

[0105] f2(x) = concat(a(X), a(y)) i-1 ), a(b i (3)

[0106] f1(x)=a(f2(x)) (4)

[0107] g i =tanh(f1(x)) (5)

[0108] In the formula, x is the input feature map, representing the current input data, X is the original feature map, and y i-1 Let b be the output feature map of the previous layer, representing the output result of the previous layer. i For the next input feature map, a() is a 3×3 convolutional layer, a(X) is the feature map obtained by passing the original feature map X through a 3×3 convolutional layer, and a(y) is the feature map obtained by passing the original feature map X through a 3×3 convolutional layer. i -1) represents the output feature map y of the previous layer. i-1 The feature map obtained after a 3×3 convolution layer, a(b) i ) represents the feature mapping b of the current input. i The feature map is obtained after a 3×3 convolution layer. `concat` is the vector concatenation function. `f2(x)` is the feature map obtained after concatenating the feature maps. `f1(x)` is the feature map obtained after a fully connected neural network layer. `tanh()` is the hyperbolic tangent function in the activation function. i The gated vector is obtained by performing a nonlinear transformation on f1(x);

[0109] Equation (3) describes the feature mapping process, where concat() represents concatenation along the feature dimension. This connection method can fuse feature information extracted at different scales to further enhance the network's expressive power.

[0110] Equation (4) describes the computation process of a fully connected neural network, where a() represents the activation function. f1(x) represents the feature map obtained through a single layer of a fully connected neural network, which will be used as the input to the gating vector.

[0111] Equation (5) describes the calculation process of the gating vector, where tanh() represents the hyperbolic tangent activation function. The gating vector is used to control the memory state and output state of the neural network at the current time step, so as to improve the network's flexibility and generalization ability.

[0112] Step 4.2.3: The data processed in step 4.2.2 is mapped to form i copies of data y. i y i After stacking, the data is fed into an l×l convolution, with the following mapping formula:

[0113]

[0114] In the formula, c i Let be the i-th input data, representing the input of the current layer, d i Let K be the i-th output feature map, representing the output result of the current layer. i For convolution operations, the input data is processed by convolution to obtain feature representations, g i Here, s is the gate vector, s is the number of layers in the neural network, and s represents the depth of the entire network.

[0115] Step 4.2.4: The data after 1×1 convolution is fed into the ECA attention module. The ECA attention module performs global average pooling on the input feature data to obtain a feature vector with 1 channel. The feature vector is then subjected to one-dimensional convolution.

[0116] Step 4.2.5: Apply the Sigmoid activation function to the vector after one-dimensional convolution to obtain a vector with values ​​between 0 and 1;

[0117] Step 4.2.6: Multiply the obtained vector element-wise with the original input feature data to obtain the weighted feature data;

[0118] Step 5: Train the network model by combining the center loss and cross-entropy loss functions with the defined optimization algorithm. The flowchart is as follows: Figure 3 As shown, the collected data is input into the trained model for human activity classification;

[0119] This invention targets the field of human activity recognition and combines the cross-entropy loss function and the center loss function. The cross-entropy loss is used to increase inter-class differences, while the center loss is used to reduce intra-class differences. After calculation, the two are summed by weight values ​​to obtain the total loss function, where the weight values ​​are hyperparameters.

[0120] Step 5.1: Add a center loss calculation layer to the network model, and use the center loss function and cross-entropy loss function to calculate the difference between sample features and intra-class centers;

[0121] The central loss function is as follows:

[0122]

[0123] In the formula, Loss crThe center loss function is used to measure the difference between sample features and class centers, where N is the number of samples and z is the center of the class. i Let be the feature vector of the i-th sample, representing the output of the network, and center be the class center, representing the center of the class to which the current sample belongs;

[0124] The cross-entropy loss function is as follows:

[0125]

[0126] In the formula, Loss ce The cross-entropy loss function measures the difference between the model output and the true label, where n is the number of samples and p(z) is the number of samples. i Let q(z) be the true label probability distribution of the i-th sample, representing the probability that the sample belongs to each category. i Let be the predicted label probability distribution of the i-th sample, representing the model's predicted probability that the sample belongs to each category;

[0127] Step 5.2: Define the optimization algorithm: When using the AdamW optimization algorithm to train the network model and find the optimal training batch, if the optimal training batch exceeds α (set to 64 in this invention), the Lion optimization algorithm is used. The optimal learning rate on the AdamW optimization algorithm is mapped to the learning rate of the Lion optimization algorithm at a ratio of β:1 (set to 3:1 in this invention), and training is based on this learning rate. Otherwise, the AdamW optimization algorithm is used for training.

[0128] Step 5.3: Use K-fold cross-validation to divide the output data of the network model obtained in Step 4 into a training set and a validation set;

[0129] Step 5.4: Train the network model using the training set: Load the center loss function and cross-entropy loss into the optimization algorithm respectively. Use the cross-entropy loss function to optimize the classification accuracy of the model, and the center loss function to optimize the clustering effect of the feature representation.

[0130] Step 5.5: Calculate the total loss function after training each batch of models and update the weight parameters of the network model through backpropagation. The formula for calculating the total loss function is as follows:

[0131] Loss total =Loss ce +λLoss cr (9)

[0132] In the formula, λ is the center loss weight, and Loss total The summation of the loss functions is used as the final loss function in model training;

[0133] Step 5.6: Evaluate the trained network model using the validation set: Calculate the values ​​of the cross-entropy loss function and the center loss function of the trained network model. Based on the classification accuracy and the value of the center loss function on the validation set, determine whether the model's performance meets the requirements. If it meets the requirements, store it as the final network model. If it does not meet the requirements, adjust the model's hyperparameters and retrain the model.

[0134] Step 5.7: Input all the data collected by the sensors in step 1 into the final network model to identify human activities;

[0135] Step 5.8: Evaluate the final model using the test set in the corresponding dataset: Calculate the classification accuracy and the value of the center loss function of the model, compare the test results with other existing methods, and verify its effectiveness.

Claims

1. A comprehensive optimization method for human activity recognition based on multi-scale metric learning, characterized in that, include: Step 1: Collect signal data about human activities, organize the data into a table, and then handle missing values; Step 2: Extract the time series data from the sensor data and preprocess the time series data; Step 3: When there is a class imbalance problem in the time series dataset after Step 2, the Borderline-SMOTE upsampling algorithm is used to oversample the boundary samples of the minority class in the dataset to obtain a dataset with a balanced number of class samples. Step 4: Construct a novel multi-scale attention mechanism gated residual deep convolutional neural network model MAG-Res2Net, and use the neural network model to extract features from the dataset; Step 4.1: Based on the ResNet module, add multi-scale filters, gating modules, and ECA attention modules in sequence to construct a new deep convolutional neural network model; Step 4.1.1: Build a ResNet module by replacing the n-channel 3×3 convolution kernels with a set of filter banks smaller than 3×3 convolution kernels, each of which has w channels, and connect the filter banks in a hierarchical manner similar to residuals to form a Res2Net module; Step 4.1.2: Add a gated module after each 3×3 convolution to form a Gated-Res2Net module; Step 4.1.3: Combine the Gated-Res2Net module and the ECA attention module to form a novel multi-scale attention mechanism gated residual deep convolutional neural network model MAG-Res2Net; Step 4.2: Feed the dataset into the established network model for feature extraction, feature combination, and prediction; Step 4.2.1: The dataset is divided into i parts after a 1×1 convolution. The first piece of data Other data besides these are fed into a 3×3 convolution; Step 4.2.2: After processing in Step 4.2.1, the gating module selects features with a threshold range of 1 ± δ for the data. The selected effective features are then mapped and passed to the next layer of the network model. The feature mapping formula includes: ； ； ； In the formula, Let X be the input feature map, representing the current input data, and let X be the original feature map. This is the output feature map of the previous layer, representing the output result of the previous layer. For the next input feature mapping, It is a 3×3 convolutional layer. The feature map is obtained by passing the original feature map X through a 3×3 convolution layer. For the output feature mapping of the previous layer The feature map obtained after one 3×3 convolution layer Feature mapping for the current input The feature map is obtained after a 3×3 convolution layer, and concat is the vector concatenation function. The feature maps are obtained by concatenating the feature maps. The feature map is obtained after passing through a fully connected neural network layer. The hyperbolic tangent function is the activation function. As a gated vector, through the Obtained by performing a nonlinear transformation; Step 4.2.3: The data processed in step 4.2.2 is mapped to form i sets of data. , After stacking, the data is fed into a 1×1 convolution, with the following mapping formula: ； In the formula, Let be the i-th input data, representing the input of the current layer. Let be the i-th output feature map, representing the output result of the current layer. The convolution operation processes the input data through convolution to obtain feature representations. For the gated vector, The number of layers in the neural network represents the depth of the entire network. Step 4.2.4: The data after 1×1 convolution is fed into the ECA attention module. The ECA attention module performs global average pooling on the input feature data to obtain a feature vector with 1 channel. The feature vector is then subjected to one-dimensional convolution. Step 4.2.5: Apply the Sigmoid activation function to the vector after one-dimensional convolution to obtain a vector with values ​​between 0 and 1; Step 4.2.6: Multiply the obtained vector element-wise with the original input feature data to obtain the weighted feature data; Step 5: Train the network model by combining the center loss and cross-entropy loss functions and optimization algorithms, and input the collected data into the trained model to classify human activities.

2. The comprehensive optimization method for human activity recognition based on multi-scale metric learning according to claim 1, characterized in that, Step 1 includes: Step 1.1: Use accelerometers, gyroscopes and wireless sensors to record sensor data during human activities, including: acceleration, angular velocity, signal delay, signal strength information, and organize the data into a table. Step 1.2: Missing human activity data is filled in using a linear interpolation algorithm.

3. The comprehensive optimization method for human activity recognition based on multi-scale metric learning according to claim 1, characterized in that, Step 2 includes: Step 2.1: Extract the time series data from the sensor data using the sliding window method; Step 2.2: Preprocess the time series data using data cleaning, data standardization, data segmentation, data labeling, and data augmentation methods.

4. The comprehensive optimization method for human activity recognition based on multi-scale metric learning according to claim 1, characterized in that, Step 3 includes: Step 3.1: Based on different datasets and training tasks, the two branches of the Borderline-SMOTE upsampling algorithm, Borderline-SMOTE1 and Borderline-SMOTE2, are used to oversample the minority class samples. Step 3.2: Verify whether the number of samples in the new minority class samples is balanced: If balanced, merge the new minority class samples with the majority class samples in the original dataset to form a balanced dataset. If unbalanced, the Borderline-SMOTE1 or Borderline-SMOTE2 branch algorithm in the Borderline SMOTE upsampling algorithm needs to be used again to generate samples to achieve the effect of balanced sample number and then form a balanced dataset.

5. The comprehensive optimization method for human activity recognition based on multi-scale metric learning according to claim 1, characterized in that, Step 5 includes: Step 5.1: Add a center loss calculation layer to the network model, using the center loss function and cross-entropy loss function to calculate the difference between sample features and within-class centers: ； In the formula, The center loss function is used to measure the difference between sample features and class centers. For the sample size, Let be the feature vector of the i-th sample, representing the output of the network. The category center represents the center of the category to which the current sample belongs; ； In the formula, Cross-entropy loss function, used to measure the difference between the model output and the true label. For the sample size, Let be the true label probability distribution of the i-th sample, representing the probability that the sample belongs to each category. Let be the predicted label probability distribution of the i-th sample, representing the model's predicted probability that the sample belongs to each category; Step 5.2: Define the optimization algorithm: When using the AdamW optimization algorithm to train the network model and find the optimal training batch, if the optimal training batch exceeds α, the Lion optimization algorithm is used. The optimal learning rate on the AdamW optimization algorithm is mapped to the learning rate of the Lion optimization algorithm in a β:1 ratio, and training is based on this learning rate. Otherwise, the AdamW optimization algorithm is used for training. Step 5.3: Use K-fold cross-validation to divide the output data of the network model into a training set and a validation set; Step 5.4: Train the network model using the training set: Load the center loss function and cross-entropy loss into the optimization algorithm respectively. Use the cross-entropy loss function to optimize the classification accuracy of the model, and the center loss function to optimize the clustering effect of the feature representation. Step 5.5: Calculate the total loss function after training each batch of models and update the weight parameters of the network model through backpropagation: ； In the formula, λ is the center loss weight. The summation of the loss functions is used as the final loss function in model training; Step 5.6: Evaluate the trained network model using the validation set: Calculate the values ​​of the cross-entropy loss function and the center loss function of the trained network model. Based on the classification accuracy and the value of the center loss function on the validation set, determine whether the model's performance meets the requirements. If it meets the requirements, store it as the final network model. If it does not meet the requirements, adjust the model's hyperparameters and retrain the model. Step 5.7: Input all the data collected by the sensors into the final network model to identify human activities.

6. The comprehensive optimization method for human activity recognition based on multi-scale metric learning according to claim 4, characterized in that, Step 3.1 includes: Step 3.1.1: Divide minority class samples into 3 categories based on the proportion of minority class samples in neighboring samples: more than half are safe samples, less than half are dangerous samples, and if there are no minority class samples, they are judged as noise. Only oversample minority class samples that exhibit the behavior of dangerous samples. Step 3.1.2: Calculate the number of dangerous minority class sample points. The distance matrix D is obtained by taking the Euclidean distance between the i-th sample point and all sample points in the minority class sample set S. D(i, p) represents the Euclidean distance between the i-th sample point and the p-th sample point. Each row of the distance matrix D is sorted, and the columns corresponding to the first k minimum values ​​are found and stored in a list. This list is the k nearest neighbors of the i-th sample point. Step 3.1.3: When the minority class samples are spatially concentrated, use Borderline-SMOTE1 to generate new samples for the danger points, and randomly select j minority class samples from the k nearest neighbors as the nearest neighbor samples. When the minority class samples are spatially dispersed, use Borderline-SMOTE2 to generate new samples for the danger points, and randomly select a sample from the k nearest neighbors as the nearest neighbor sample, without paying attention to the sample category. Step 3.1.4: Calculate the dangerous minority class samples Each sample of its chosen nearest neighbor The difference vector is calculated using the following formula: ； In the formula, For the i-th dangerous minority class sample, For the i-th minority class sample, the j-th nearest neighbor sample is... This is the difference vector between minority class samples and nearest neighbor samples. When using Borderline-SMOTE2, j=1. Step 3.1.5: Synthesize new minority class samples, using the following formula: = ； In the formula, For new minority class samples, This is the difference vector between the minority class sample and its nearest neighbor sample. It is a constant.

Images