A multi-angle joint activity recognition classification method based on local loss

Abstract

The application discloses a multi-angle joint activity recognition classification method based on local loss, and belongs to the field of activity recognition classification.The application realizes the method by weighting and fusing a local similarity matching loss function and a local focal loss function; in view of the difficult classification characteristics of few sample number activity categories, the loss value of the few sample activity categories is calculated and improved during each training, so that the attention degree of a benchmark model to the few sample categories is increased, and then the inter-class sample imbalance problem in the field of activity recognition is improved.The application realizes the forming process of the decision boundary of the benchmark model by block training of three convolutional layers and one fully connected layer, the local similarity matching loss function is used for the output of the convolutional layer, the local focal loss function is used for the output of the local linear connection layer matched with the convolutional layer, the model parameter lightweight can be realized under the premise of ensuring the activity recognition classification performance, and the hardware memory reuse rate in the model training process is improved.

Description

Technical Field

[0001] This invention belongs to the field of activity recognition and classification, and particularly relates to a multi-angle joint activity recognition and classification method based on local loss. Background Technology

[0002] Activity recognition utilizes time-series data generated by sensors during motion to process and analyze it, identifying the category of the activity being performed. It has been widely applied in fields such as military equipment, smart healthcare, and industrial monitoring. The purpose of activity recognition classification is to categorize time-series data into a specific activity category based on the characteristics of the obtained samples, and then apply the classification to corresponding scenarios.

[0003] With the widespread use of deep learning technology in recent years, deep learning methods, primarily Convolutional Neural Networks (CNNs), have received considerable attention and research in the field of activity recognition and classification, achieving significant progress. While various CNN-based activity recognition methods have achieved good classification performance, these methods typically use only a global loss function to train the model, neglecting the problem of class imbalance in the dataset. This results in CNN models based on global loss functions lacking specificity for addressing the imbalance problem of activity classes; that is, they place less emphasis on classes with fewer samples, failing to effectively classify those activity classes and often exhibiting overfitting to activity classes with larger sample sizes, thus failing to establish optimal activity decision boundaries. Furthermore, traditional global loss functions require a large number of trainable parameters. Even after the model has completed the forward propagation of the data flow in the first half, the hardware memory must still retain the weight parameters of the entire model during the forward and backward propagation processes, thus the backward locking phenomenon hinders memory reuse. Therefore, designing a multi-angle joint loss function with high training efficiency and lightweight design is of great practical significance for the field of activity recognition and classification. Summary of the Invention

[0004] The purpose of this invention is to address the problems of insufficient attention to few-sample categories and low memory reuse in existing activity recognition methods. This invention provides a multi-angle joint activity recognition and classification method based on local loss. By weighted fusion of the local similarity matching loss function and the local focus loss function, a baseline model centered on three convolutional layers and one fully connected layer, with each convolutional layer paired with a corresponding local linear connection layer, forms a complex decision boundary between highly variable similar activity classes and highly similar different activity classes. Addressing the difficulty in classifying few-sample activity categories, the loss value for few-sample activity classes is calculated and increased during each training iteration to enhance the baseline model's attention to these categories, thereby improving the inter-class sample imbalance problem in activity recognition. Furthermore, this invention achieves the formation of the baseline model's decision boundary by training the three convolutional layers and one fully connected layer in blocks. The local similarity matching loss function is used for the output of the convolutional layers, and the local focus loss function is used for the output of the local linear connection layer paired with the convolutional layers. This allows for lightweight model parameters while maintaining activity recognition and classification performance, accelerating the training speed of the baseline model and improving hardware memory reuse during model training.

[0005] The objective of this invention is achieved through the following technical solution.

[0006] This invention discloses a multi-angle joint activity identification method based on local loss, comprising the following steps:

[0007] Step 1: Design a local similarity matching loss function module; This module performs scaling on the output of each convolutional layer and calculates the similarity matrix S between the output feature map and the sample labels. F S Y And compare S F and S Y The mean squared error is used to obtain the local similarity matching loss, match.

[0008] match = MSE(S F S Y )

[0009] The method for scaling the output of each convolutional layer is as follows: The 3D feature map of the convolutional layer is scaled... Reduce the dimensionality of the data to 2D to obtain the feature map.

[0010] Calculate the similarity matrix S of the output feature map F The method is as follows: For the feature map F after scaling transformation... 2D Calculate the mean and variance, then subtract the mean from the feature map, and divide by the variance to obtain the normalized result F. 2D_normalizaitonThen for F 2D_normalizaiton Perform matrix transpose to obtain F 2D_normalizaiton_t , will F 2D_normalizaiton With F 2D_normalizaiton_t Performing matrix multiplication yields the similarity matrix S of the feature maps. F .

[0011] Calculate the similarity matrix S of the sample labels Y The method is as follows: First, for sample labels Y∈R batch Perform one-hot encoding to obtain the one-hot code label Y_onehot∈R batch×n Next, the mean and variance of the one-hot code label Y_onehot are calculated, and normalization is performed by subtracting the mean and dividing the variance to obtain Y_onehot_normalization. Then, the matrix transpose of Y_onehot_normalization is performed to obtain Y_onehot_normalization_t. Finally, Y_onehot_normalization and Y_onehot_normalization_t are multiplied by matrix to obtain the similarity matrix S of the sample labels. Y .

[0012] Step 2: Design the local focus loss function module; calculate the local focus loss (focus) between the predicted output of the local linear connection layer and the sample labels using the local focus loss function module.

[0013] focus = FL(Y_local, Y)

[0014] Step 3: Design a multi-angle joint loss function module; The local similarity matching loss (match) and the local focus loss (focus) are weighted and fused using this module to obtain the total loss (focmat).

[0015] focmat=α*match+(1-α)*focus

[0016] Where α is a weighting coefficient, which was set experimentally.

[0017] Step 4: Read the activity data to be classified and construct the activity dataset; divide the dataset into training set, validation set and test set;

[0018] Step 5: Construct a baseline model. The baseline model centers on three convolutional layers and one fully connected layer, with each convolutional layer paired with a corresponding locally linearly connected layer. The three convolutional layers are responsible for extracting spatial features from the activity dataset described in Step 4, obtaining spatial feature vectors. The fully connected layer is responsible for predicting the spatial feature vectors extracted by the three convolutional layers, resulting in the activity classification result and completing the activity recognition task. Applying the multi-angle joint loss function module from step three to the baseline model, the spatial feature vector output by each convolutional layer is not only fed into subsequent connected convolutional or fully connected layers, but also into the corresponding local linear connection layer. The local linear connection layer is responsible for predicting the spatial feature vector calculated by the convolutional layer. The total loss focmat mentioned in step three can be directly calculated using the output of the convolutional layer and the predicted output of the local linear connection layer. This allows each convolutional layer to perform block-based training of the baseline model after it has completed its own convolutional calculation, without waiting for the entire baseline model to finish its calculation. This reduces the number of trainable parameters in the activity recognition process and improves the reusability of hardware memory.

[0019] Step Six: Model Training; Input the training and validation set data from Step Four into the model constructed in Step Five. The first convolutional layer performs convolution operations on the active samples in the training set described in Step Four, extracting low-level spatial feature vectors from the active samples. These extracted spatial feature vectors are then used as input to the local linear connection layer corresponding to the second and first convolutional layers. The second convolutional layer performs another convolution operation on the output of the first convolutional layer to obtain a more abstract spatial feature vector, which is then fed into the third convolutional layer and the local linear connection layer corresponding to the second convolutional layer. The third convolutional layer further performs convolution extraction and dimensionality refinement on the spatial feature vector output from the second convolutional layer, and then feeds the output into the corresponding local linear connection layer and fully connected layer. The low-dimensional spatial feature output with a large receptive field obtained after passing through three convolutional layers is used as input to the fully connected layer, which is responsible for predicting and classifying the output.

[0020] Each convolutional layer obtains its own spatial feature vector through convolution. Based on the local similarity matching loss function module in step one, the similarity matrix between the output of each convolutional layer and the sample label is calculated, and the mean squared error of both is calculated to obtain the local similarity matching loss value `match`. This loss value represents the fitting ability of each convolutional layer between the active sample and the sample label. The weight parameter matrix is ​​optimized through backpropagation, enabling the convolutional layer to better fit different active samples and form a better decision boundary. The corresponding local linear connection layer of each convolutional layer predicts the output of that convolutional layer to obtain the predicted output. According to the local focus loss function module in step two, the local focus loss `focus` is calculated and obtained. This loss value reflects the degree of difference between the predicted output of the local linear connection layer and the sample label. For a few-sample active class, the calculated value is... The focus value is relatively large. For activity categories with a large number of samples, the calculated focus value is relatively small. Through the backpropagation mechanism, the local linear connection layer and the corresponding convolutional layer can increase their attention to activity categories with fewer samples. The weight parameter matrix will be more sensitive to activity categories with fewer samples, and the decision boundary will be more biased towards activity categories with fewer samples. According to the multi-angle joint loss function module in step three, the local similarity matching loss value match and the local focus loss focus obtained above are weighted and fused to serve as the total loss value focmat of the convolutional layer and the local linear connection layer. This loss value can enable the convolutional layer and the local linear connection layer to improve the recognition performance of activity categories with fewer samples while ensuring the accuracy of activity recognition classification. This effectively improves the sample imbalance problem in the field of activity recognition and completes the training of the benchmark model.

[0021] Step 7: Model Testing; Input the activity samples from the test set data into the baseline model trained in Step 6, and output the predicted label for each activity sample. Use the output predicted label as the activity category corresponding to the sample to complete the classification. While ensuring the overall activity recognition accuracy, improve the classification performance of the baseline model for activity categories with fewer samples, and improve the sample imbalance problem in the activity recognition field.

[0022] Beneficial effects:

[0023] 1. To address the problem of biased predictions caused by the sensitivity of global loss functions to activity class imbalance, this invention discloses a multi-angle joint activity recognition and classification method based on local loss. This method includes a local similarity matching loss function module and a local focus loss function module. The local similarity matching loss function allows convolutional layers and corresponding local linear connection layers to effectively fit the activity sample data, forming a better classification decision boundary and achieving better overall classification performance. The local focus loss function enables convolutional layers and corresponding local linear connection layers to effectively capture and learn the spatial features of fewer activity classes, improving the focus on fewer activity classes. Through... The weighted fusion of local similarity matching loss function and local focus loss function can, while ensuring overall classification performance, allow the benchmark model to assign different importance to different activity classes, forming a complex decision boundary and improving classification performance for imbalanced sample problems. Simultaneously, this invention introduces local loss into the loss function, replacing the global loss function, and implements separate training for different network layers. During forward and backward propagation, the parameters of each layer can be updated without waiting for the entire benchmark model to complete training. This reduces the number of trainable parameters in the benchmark model while maintaining classification performance, achieving a lightweight model, accelerating model training, and improving hardware memory reuse.

[0024] 2. This invention provides a multi-angle joint loss function method based on local loss. The local similarity matching loss function reduces the dimensionality of the output feature map F of each convolution operation to 2D and performs one-hot encoding on the sample labels. Normalization is performed by calculating the mean and variance of the dimensionality-reduced feature map and the one-hot encoded sample labels. The two normalized matrices are then transposed and multiplied to obtain the pairwise similarity between the samples in the convolution feature map and the pairwise similarity between the sample labels Y, i.e., their respective similarity matrices S. The similarity matching distance is then calculated, which enables the convolutional layer and the corresponding local linear connection layer to learn better spatial features and form a better decision boundary.

[0025] 3. This invention provides a multi-angle joint loss function method based on local loss. The local focus loss function optimizes the local linear connection layers of each local convolutional module. Compared with the cross-entropy loss function (CE), this focus loss function can reduce the loss value for more confident predictions and increase the loss value for less confident predictions. It can achieve hardware memory reuse and improve memory reuse rate while ensuring that the convolutional layer and the corresponding local linear connection layer learn better spatial features.

[0026] 4. This invention provides a multi-angle joint loss function method based on local loss. It adopts the idea of ​​local training, dividing the network model into layers as local modules for training. This can reduce the hardware overhead of storing weight parameters during forward and backward propagation, improve the hardware memory reuse rate, and achieve lightweighting of the network model. Attached Figure Description

[0027] Figure 1 A schematic diagram illustrating the principle of the multi-angle joint activity recognition and classification method based on local loss provided by this invention;

[0028] Figure 2 A flowchart illustrating the training process of the multi-angle joint activity recognition and classification method based on local loss provided by this invention;

[0029] Figure 3 The confusion matrix diagram of the classification results of the UCIHAR dataset provided by the present invention is shown; where (a) is the loss function using global cross-entropy, (b) is the loss function using local similarity matching, (c) is the loss function using local focus, and (d) is the loss function using multi-angle joint loss.

[0030] Figure 4 The confusion matrix diagram of the classification results of the WISDM dataset provided by the present invention is shown; where (a) is the loss function using global cross-entropy, (b) is the loss function using local similarity matching, (c) is the loss function using local focus, and (d) is the loss function using multi-angle joint loss.

[0031] Figure 5 The diagram shows the confusion matrix of the classification results of the Unimib-SHAR dataset provided by this invention; where (a) is the loss function using global cross-entropy, (b) is the loss function using local similarity matching, (c) is the loss function using local focus, and (d) is the loss function using multi-angle joint loss.

[0032] Figure 6 The following are the curves showing the changes of each loss function during the training process provided by this invention: (a) is the loss change graph of each loss function on the UCI HAR dataset, (b) is the loss change graph of each loss function on the WISDM dataset, and (c) is the loss change graph of each loss function on the Unimib-SHAR dataset. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0034] like Figure 1The activity recognition and classification method based on local loss and multi-angle joint loss function disclosed in this embodiment is implemented based on a local similarity matching loss function module, a local focus loss function module, and a multi-angle joint loss function module. To ensure the universality of each loss function module, the most widely used three-layer convolutional model—each of which includes convolution, batch normalization, and activation function operations in three convolutional layers, and finally a fully connected layer—is used as the benchmark model for illustration. Figure 2 The activity identification and classification method based on multi-angle joint local loss disclosed in this embodiment has the following specific implementation steps:

[0035] Step 1: Design the local similarity matching loss function module; this module is used to scale the output of each convolutional layer and calculate the similarity matrix S between the output feature map and the sample label. F S Y And compare S F and S Y The mean squared error is used to obtain the local similarity matching loss, match.

[0036] The method for scaling the output of each convolutional layer is as follows: for each sample input X∈R 1×c×L Where C is the number of sensor channels and L is the sequence length, the feature map corresponding to the convolutional layer is obtained after passing through the convolutional layer. C c Where H is the number of output channels of the convolutional layer, and H and W are the width and height of the feature map, respectively. Their mathematical expressions are as follows:

[0037] F=δ(Θ(Conv(W c ,X)))

[0038] Where δ(·) and Θ(·) represent the ReLU activation function and batch normalization operation, respectively, and Conv(·) represents the 2D convolution operation. These are the weight parameters of the convolutional layer Conv(·), C c This represents the number of output channels, i.e., the number of features; the feature map obtained above... Reduce the dimensionality of the data to 2D to obtain the feature map.

[0039] Calculate the similarity matrix S of the output feature map F The method is as follows: First, the feature map F after scaling transformation is processed. 2D Calculate the mean and variance, then subtract the mean from the feature map, and divide by the variance to obtain the normalized result F. 2D_normalizaiton Then for F 2D_nomalizaiton Perform matrix transpose to obtain F 2D_normalizaiton_t Ultimately, F 2D_nomalizaiton With F 2D_nomalizaiton_tPerforming matrix multiplication yields the similarity matrix S of the feature maps. F .

[0040] Calculate the similarity matrix S of the sample labels Y The method is as follows: First, for sample labels Y∈R batch Perform one-hot encoding to obtain the one-hot code label Y_onehot∈R batch×n Here, n is the number of activity categories corresponding to the dataset. The mean and variance of the one-hot code labels Y_onehot are then calculated, followed by normalization by subtracting the mean and dividing by the variance to obtain Y_onehot_normalization. Y_onehot_normalization is then transposed to obtain Y_onehot_normalization_t. Finally, Y_onehot_normalization and Y_onehot_normalization_t are multiplied together to obtain the similarity matrix S of the sample labels. Y .

[0041] Finally, calculate the two similarity matrices S mentioned above. F and S Y The mean squared error value is used to obtain the local similarity matching loss value, match, whose data expression is as follows:

[0042] match = MSE(S F S Y )

[0043] Where MSE(·) is the mean square error function.

[0044] Step 2: Design the local focus loss function module; this module is used to calculate the focus loss value between the predicted output of the local linear connection layer and the true label, and then optimize the weight parameter matrix of the convolutional layer and the corresponding local linear connection layer to achieve attention and focus on the few-sample class data.

[0045] It should be noted that, in order to reduce the number of trainable parameters during training and to enable backpropagation and gradient update immediately after each convolutional layer completes its operation, thus realizing the idea of ​​local training, a local linear connection layer needs to be added for prediction, and the predicted output Y_local∈R is obtained. batch×n Its data expression is as follows:

[0046] Y_local = Linear local (W F_local ,F′)

[0047] Among them, Linear local It is the locally linearly connected layer corresponding to the convolutional layer. This is the weight parameter matrix of the local linear connection layer, and the input is the result of the two-dimensional scaling transformation of the feature map F. After obtaining the predicted output Y_local, the focus loss between it and the true sample label Y is calculated, and its mathematical expression is as follows:

[0048] focus = FL(Y_local, Y)

[0049] FL(·) is the focus function, and its formula is as follows:

[0050]

[0051] Where, p t It is the predicted probability value for each sample in Y_local, s p It is the class score for correctly predicting activity category n, where γ is a hyperparameter. Compared to the cross-entropy loss function, the focus loss function can reduce the loss value of more confident predictions, while increasing the loss value of less confident predictions, which are often activity categories with fewer samples. This makes the model pay attention to activity categories with fewer samples, forming a complex decision boundary between the same semantic class with high variability and different semantic classes with high similarity.

[0052] Step 3: Design a multi-angle joint loss function module. This module is used to weightedly fuse the local similarity matching loss (match) and the local focus loss (focus), and the fused loss value is used as the total loss value (focmat) of the local convolutional layer and the local linear connection layer. Its mathematical expression is as follows:

[0053] focmat=α*match+(1-α)*focus

[0054] It should be noted that α is a weighting coefficient, set experimentally to achieve local training and local backpropagation. For each convolutional layer and its corresponding locally linearly connected layer, gradient updates begin after calculating the focal length. When the input reaches the final fully connected layer after forward propagation, the final predicted output P∈R is obtained. batch×n At this point, only the local focus loss function is calculated to update the weight parameters of the fully connected layer and complete one training cycle.

[0055] Step 4: Read the activity data to be classified and construct the activity dataset; perform sliding window sampling on the activity data of C sensor channels with a spatial size of 128 points to obtain N activity data samples with a dimension of C×128, and increase the dimension of each sample to make it a 1×C×128 shape, which together form the sample set; randomly sample the sample set according to the ratio of 7:1:2, and divide 70% of the labeled samples into the training set, 10% into the validation set, and 20% into the test set;

[0056] Step 5: Construct a baseline model. The baseline model centers on three convolutional layers and one fully connected layer, with each convolutional layer paired with a corresponding locally linearly connected layer. The three convolutional layers are responsible for extracting spatial features from the activity dataset described in Step 4, obtaining spatial feature vectors. The fully connected layer is responsible for predicting the spatial feature vectors extracted by the three convolutional layers, resulting in the activity classification result and completing the activity recognition task. By applying the multi-angle joint loss function module from step three to the baseline network, the spatial feature vector output by each convolutional layer is not only fed into subsequent connected convolutional or fully connected layers, but also into the corresponding local linear connection layer. The local linear connection layer is responsible for predicting the spatial feature vector calculated by the convolutional layer. The total loss focmat mentioned in step three can be directly calculated using the output of the convolutional layer and the predicted output of the local linear connection layer. This allows each convolutional layer to optimize its weight parameter matrix and the corresponding local linear connection layer after its own convolution calculation without waiting for the entire baseline model to finish. This completes the block-based training of the baseline model, thereby reducing the number of trainable parameters in the activity recognition process and improving the reusability of hardware memory.

[0057] Step Six: Model Training; Input the training and validation set data from Step Four into the model constructed in Step Five. The first convolutional layer performs convolution operations on the active samples in the training set described in Step Four, extracting low-level spatial feature vectors from the active samples. These extracted spatial feature vectors are then used as input to the local linear connection layer corresponding to the second and first convolutional layers. The second convolutional layer performs another convolution operation on the output of the first convolutional layer to obtain a more abstract spatial feature vector, which is then fed into the third convolutional layer and the local linear connection layer corresponding to the second convolutional layer. The third convolutional layer further performs convolution extraction and dimensionality refinement on the spatial feature vector output from the second convolutional layer, and then feeds the output into the corresponding local linear connection layer and fully connected layer. Finally, the low-dimensional spatial feature output with a large receptive field obtained after passing through three convolutional layers is used as input to the fully connected layer, which is responsible for predicting and classifying the output.

[0058] Each convolutional layer obtains its own spatial feature vector through convolution. Based on the local similarity matching loss function module in step one, the similarity matrix between the output of each convolutional layer and the sample label is calculated, and the mean squared error of both is calculated to obtain the local similarity matching loss value `match`. This loss value represents the fitting ability of each convolutional layer between the active sample and the sample label. Optimizing the weight parameter matrix through backpropagation allows the convolutional layer to better fit different active samples, forming a better decision boundary. The corresponding local linear connection layer of each convolutional layer predicts the output of that convolutional layer to obtain the predicted output. Based on the local focus loss function module in step two, the local focus loss `focus` is calculated. This loss value reflects the degree of difference between the predicted output of the local linear connection layer and the sample label. For a few-sample active class, the calculation... The calculated focus value is relatively large. For activity categories with a large number of samples, the calculated focus value is relatively small. Through the backpropagation mechanism, the local linear connection layer and the corresponding convolutional layer can increase their attention to activity categories with fewer samples. The weight parameter matrix will be more sensitive to activity categories with fewer samples, and the decision boundary will be more biased towards activity categories with fewer samples. According to the multi-angle joint loss function module in step three, the local similarity matching loss value match and the local focus loss focus obtained above are weighted and fused to serve as the total loss value focmat of the convolutional layer and the local linear connection layer. This loss value can improve the recognition performance of the convolutional layer and the local linear connection layer for activity categories with fewer samples while ensuring the accuracy of activity recognition classification. This effectively improves the problem of fewer sample categories in the field of activity recognition and completes the training.

[0059] Step 7: Model Testing; Input the activity samples from the test set data into the baseline model trained in Step 6, and output the predicted label for each activity sample. Use the output predicted label as the activity category corresponding to the sample to complete the classification. While ensuring the overall activity recognition accuracy, this method improves the classification performance of the baseline model for activity categories with few samples, effectively addressing the sample imbalance problem in the activity recognition field. This invention was used to classify three publicly available activity datasets: UCI HAR, WISDM, and Unimib-SHAR. The classification accuracy (Acc) and F1 coefficient (F1-score) for different datasets are shown in Table 1. It can be seen that this invention achieves overall classification accuracy of 95.92%, 93.34%, and 75.63% on the three publicly available activity datasets: UCI HAR, WISDM, and Unimib-SHAR, respectively.

[0060] like Figure 3As shown in (d), on the UCI HAR dataset, this invention can accurately identify and classify simple activity categories such as "Walking" and "Lying", with only 5 activity samples being misclassified. For the classification of similar activities such as "Walking Upstairs" and "Walking Downstairs", only 15 activity samples were misclassified, achieving high-precision classification. For the difficult-to-distinguish activity categories such as "Sitting" and "Standing", it also achieved good identification and classification, with a relatively small number of misclassified samples.

[0061] like Figure 4 As shown in (d), on the WISDM dataset, this invention can achieve high-precision classification for categories with a large number of active samples, such as "Walking" and "Jogging". For difficult-to-classify categories with a small number of active samples, such as "Sitting" and "Standing", there are fewer misclassified samples.

[0062] like Figure 5 As shown in (d), on the Unimb-SHAR dataset, this invention can achieve fewer misclassifications for categories with a large number of activity samples, such as "Walking", "Running" and "Going Down". For activities with a small number of activity samples, such as "Standing Up from Sitting" and "Standing Up from Laying", the number of misclassified samples is also small.

[0063] like Figure 6 As shown in (a), (b), and (c), on the three publicly available activity datasets—UCI HAR, WISDM, and Unimib-SHAR—the loss value of this invention does not exhibit significant fluctuations or changes during training; the loss value curve is smoother, and the loss value is lower. Therefore, it can be demonstrated that, while achieving high-precision activity recognition, this invention can effectively improve the inter-class sample imbalance phenomenon in the field of activity recognition, effectively improve the recognition accuracy for activity categories with fewer samples, and make the training process more stable.

[0064] Finally, as shown in Table 1, the number of trainable parameters (Params) of this invention on the three public active datasets UCI HAR, WISDM, and Unimib-SHAR are 0.92*10^5, 0.97*10^5, and 1.44*10^5, respectively. This indicates that this invention can effectively reduce the number of parameters in the training process and improve the hardware memory reuse rate while achieving high classification accuracy.

[0065] Table 1: Classification results and parameter counts for different datasets

[0066]

[0067] Comparative example:

[0068] like Figure 3 As shown in (a), (b), and (c), comparative experiments were conducted on the UCI HAR dataset using the global cross-entropy loss function, the local similarity matching loss function, and the local focus loss function, respectively, to obtain classification results. For simple activity categories, such as "Walking," all three methods achieve good classification results. However, for activity categories that are difficult to distinguish, such as "Sitting" and "Standing," the number of misclassified samples is relatively large, with too many activity samples being incorrectly classified.

[0069] like Figure 4 As shown in (a), (b), and (c), comparative experiments were conducted on the WISDM dataset using the global cross-entropy loss function, the local similarity matching loss function, and the local focus loss function, respectively, to obtain classification results. For categories with a large number of activity samples, all three methods can achieve good classification results. However, for activity categories with a small number of activity samples, such as "Sitting" and "Standing," good classification cannot be achieved, and the number of misclassified samples is relatively large.

[0070] like Figure 5 As shown in (a), (b), and (c), comparative experiments were conducted on the Unimib-SHAR dataset using the global cross-entropy loss function, the local similarity matching loss function, and the local focus loss function, respectively, to obtain classification results. For difficult-to-classify activity categories with a small number of activity samples, such as "StandingUp from Sitting" and "Standing Up from Laying," all three methods often fail to achieve good classification, with many samples being misclassified as "Lying Down from Standing." This indicates that the methods cannot achieve good classification and recognition for activity categories with few samples and cannot effectively improve the imbalance of samples between classes.

[0071] like Figure 6 As shown in (a)(b)(c), comparative experiments were conducted on three publicly available activity datasets—UCI HAR, WISDM, and Unimib-SHAR—using the global cross-entropy loss function, the local similarity matching loss function, and the local focus loss function, respectively. The results show that the global cross-entropy loss function exhibits large fluctuations and a lack of stability during training. While the local similarity matching loss function and the local focus loss function are relatively stable during training, their loss values ​​are high, indicating that using only one loss function cannot achieve good recognition and classification.

[0072] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-angle joint activity recognition method based on local loss, characterized in that: Includes the following steps, Step 1: Design a local similarity matching loss function module; This module performs scaling on the output of each convolutional layer and calculates the similarity matrix S between the output feature map and the sample labels. F S Y And compare S F and S Y The mean squared error is used to obtain the local similarity matching loss, match. match=MSE(S F ,S Y ) The method for scaling the output of each convolutional layer is as follows: The 3D feature map of the convolutional layer is scaled... Reduce the dimensionality of the data to 2D to obtain the feature map. Calculate the similarity matrix S of the output feature map F The method is as follows: For the feature map F after scaling transformation... 2D Calculate the mean and variance, then subtract the mean from the feature map, and divide by the variance to obtain the normalized result F. 2D_normalizaiton Then for F 2D_normalizaiton Perform matrix transpose to obtain F 2D_normalizaiton_t , will F 2D_normalizaiton With F 2D_normalizaiton_t Performing matrix multiplication yields the similarity matrix S of the feature maps. F ; Calculate the similarity matrix S of the sample labels Y The method is as follows: First, for sample labels Y∈R batch Perform one-hot encoding to obtain the one-hot code label Y_onehot∈R batch×n Next, calculate the mean and variance of the one-hot code label Y_onehot, and perform normalization by subtracting the mean and dividing the variance to obtain Y_onehot_normalization. Then, transpose Y_onehot_normalization to obtain Y_onehot_normalization. _ t, will eventually combine Y_onehot_normalization and Y_onehot_normalization _ Perform matrix multiplication on t to obtain the similarity matrix S of the sample labels. Y ; Step 2: Design the local focus loss function module; calculate the local focus loss (focus) between the predicted output of the local linear connection layer and the sample labels using the local focus loss function module. focus = FL(Y_local, Y) Step 3: Design a multi-angle joint loss function module; The local similarity matching loss (match) and the local focus loss (focus) are weighted and fused using this module to obtain the total loss (focmat). focmat=α*match+(1-α)*focus Where: α is the weighting coefficient, which was set through multiple experiments; Step 4: Read the activity data to be classified and construct the activity dataset; divide the dataset into training set, validation set and test set; Step 5: Construct a baseline model. The baseline model is centered around three convolutional layers and one fully connected layer, with each convolutional layer paired with a corresponding local linear connection layer. The three convolutional layers are responsible for extracting spatial features from the activity dataset described in Step 4 to obtain spatial feature vectors. The fully connected layer is responsible for predicting the spatial feature vectors extracted by the three convolutional layers to obtain the activity classification result and complete the activity recognition task. Applying the multi-angle joint loss function module from Step 3 to the baseline model, the spatial feature vectors output by each convolutional layer will not only be fed into subsequent connected convolutional or fully connected layers, but also into the corresponding local linear connection layer. The local linear connection layer is responsible for predicting the spatial feature vectors calculated by the convolutional layer. The total loss focmat described in Step 3 can be directly calculated using the output of the convolutional layer and the predicted output of the local linear connection layer. This allows each convolutional layer to perform block-based training of the baseline model without waiting for the entire baseline model to finish its calculation. Instead, it can optimize the weight parameter matrices of the convolutional layer and the corresponding local linear connection layer after completing its own convolution calculation. Step Six: Input the training and validation set data from Step Four into the model constructed in Step Five. The first convolutional layer is responsible for performing convolution operations on the active samples in the training set described in Step Four, extracting low-level spatial feature vectors from the active samples, and using the extracted spatial feature vectors as input to the local linear connection layer corresponding to the second and first convolutional layers. The second convolutional layer performs another convolution operation on the output of the first convolutional layer to obtain a higher-level abstract spatial feature vector, which is then fed into the third convolutional layer and the local linear connection layer corresponding to the second convolutional layer as input. The third convolutional layer further performs convolution extraction and dimensionality refinement on the spatial feature vector output from the second convolutional layer, and sends the output to the local linear connection layer and the fully connected layer corresponding to the third layer. The low-dimensional spatial feature output with a large receptive field obtained after passing through three convolutional layers is used as input to the fully connected layer, which is responsible for predicting the output and classifying it. Step 7: Input the activity samples from the test set data into the baseline model trained in Step 6, and output the predicted label for each activity sample. Use the output predicted label as the activity category corresponding to the sample to complete the classification. While ensuring the overall activity recognition accuracy, improve the classification performance of the baseline model for activity categories with few samples, and improve the sample imbalance problem in the activity recognition field.

2. The activity recognition method based on local loss and multi-angle joint analysis as described in claim 1, characterized in that: In step six, Each convolutional layer obtains its own spatial feature vector through convolution. Based on the local similarity matching loss function module in step one, the similarity matrix between the output of each convolutional layer and the sample label is calculated, and the mean squared error of both is calculated to obtain the local similarity matching loss value `match`. This loss value represents the fitting ability of each convolutional layer between the active sample and the sample label. The weight parameter matrix is ​​optimized through backpropagation, enabling the convolutional layer to better fit different active samples and form a better decision boundary. The corresponding local linear connection layer of each convolutional layer predicts the output of that convolutional layer to obtain the predicted output. Based on the local focus loss function module in step two, the local focus loss `focus` is calculated and obtained. This loss value reflects the predicted output of the local linear connection layer. The degree of difference between the output and the sample label is determined; through the backpropagation mechanism, the local linear connection layer and the corresponding convolutional layer increase the attention to the few-sample activity category, the weight parameter matrix becomes more sensitive to the few-sample activity category, and the decision boundary becomes more biased towards the few-sample activity category; according to the multi-angle joint loss function module in step three, the obtained local similarity matching loss value match and local focus loss focus are weighted and fused to serve as the total loss value focmat of the convolutional layer and the local linear connection layer. This loss value enables the convolutional layer and the local linear connection layer to improve the recognition performance of the few-sample activity category while ensuring the accuracy of activity recognition classification, effectively improving the sample imbalance problem in the activity recognition field, and completing the training of the benchmark model.

Images