Long-tail activity identification method for mobile dual-decoupled networks

Abstract

The present invention discloses a method for long-tail activity recognition using a dual-decoupled network on a mobile terminal, which specifically includes the following steps: collecting action signals and obtaining a training set; inputting the training set into a dual-decoupled network model, and realizing long-tail activity recognition through a mobile wearable recognition device; wherein the dual-decoupled network model is constructed by a neural network and is decoupled in a training phase and a test phase, respectively. The present invention enhances the feature extraction capability of the model by decoupling the training phase, i.e., separating feature extraction and activity classification, and adopts a multi-branch topology structure. At the same time, weight fine-tuning in the classifier phase can effectively improve the performance of the model in recognizing long-tail distributions; in the present invention, the decoupling of the training phase and the test phase significantly reduces the amount of model parameters, helps to reduce the hardware requirements of wearable recognition devices, and improves the real-time performance of activity recognition.

Description

Technical Field

[0001] The present invention belongs to the field of sensor activity recognition, and in particular to a method for recognizing long-tail activities in a dual-decoupled network on a mobile terminal. Background Art

[0002] In the current technological context, with the development of deep learning and sensor technology, the research on daily life activity recognition has made rapid progress. However, real-life activity datasets often follow a long-tail distribution, where a few dominant categories occupy most of the data, while most categories - the minority - contain only a limited number of data samples. Unsurprisingly, long-tail activity data makes rare activity types more difficult to repeat and collect than common categories (e.g., walking and leaning forward). Therefore, deep models trained using long-tail distribution datasets will inevitably be biased against common categories, i.e., instance-rich categories dominate the training process, resulting in higher recognition accuracy for these categories and lower recognition accuracy for instance-scarce types. However, in the task of sensor activity recognition, improving the recognition performance of all categories (including both instance-rich and instance-scarce categories) has not received enough attention.

[0003] Therefore, there is an urgent need for a mobile dual-decoupled network long-tail activity recognition method to address the shortcomings of the existing technology. Summary of the invention

[0004] The purpose of the present invention is to propose a mobile terminal dual decoupled network long-tail activity recognition method, which adopts a two-stage dual decoupled network structure and uses cross entropy and category balance loss function to optimize feature extraction and classification respectively during the training process.

[0005] To achieve the above object, the present invention provides a method for identifying long-tail activities in a dual-decoupled network on a mobile terminal, which specifically includes the following steps:

[0006] Collect action signals and obtain training sets;

[0007] Inputting the training set into a double decoupled network model, and realizing long-tail activity recognition through a mobile wearable recognition device;

[0008] The double decoupled network model is constructed by a neural network and is decoupled in the training phase and the testing phase respectively.

[0009] Optionally, collecting the action signal and obtaining the training set includes:

[0010] The motion signal is collected by the mobile wearable recognition device, the motion signal is preprocessed and data is divided, the training set is obtained, and the corresponding motion type is marked according to the timestamp of the synchronously recorded acquisition video.

[0011] Optionally, preprocessing and data division of the action signal includes:

[0012] Performing filtering, noise reduction and normalization processing on the motion signal to obtain a preprocessed motion signal;

[0013] Collecting the sampling frequency of the sensor device;

[0014] The time window, sliding window and data overlap rate of the action signal are set according to the sampling frequency, and the pre-processed action signal is divided into data.

[0015] Optionally, the double decoupled network model includes a convolution block with a three-layer topology structure and a fully connected classification output layer, wherein each of the convolution blocks respectively includes a convolution layer, a batch normalization layer, and adopts a ReLU activation function, wherein the convolution block is used to extract feature information of activity data in the training phase and is used for forward reasoning in the testing phase; the fully connected classification output layer is used to output classification.

[0016] Optionally, the calculation method in the training phase includes minimization and fine-tuning of weight parameters, which are respectively minimization of class balance error function, fine-tuning of L2 penalty term, and fine-tuning of MaxNorm restriction term;

[0017] The class balance error function minimization form is:

[0018] W * =argminCB(f(x i ;w),y i )

[0019] The functional form of the L2 penalty term fine-tuning is:

[0020]

[0021] The function form of the MaxNorm limit item fine-tuning is:

[0022]

[0023] Among them, W * is the overall weight parameter of the fully connected layer, argmin is the function to obtain the minimum value, CB represents the class balance error function, f is the mapping function of the fully connected layer, x i is the activity data matrix for each batch, w is the corresponding x i The weight parameter, y i is the true label, k is a hyperparameter that controls the attenuation, and st means that the second norm of the weight w should be less than the threshold r 2 value.

[0024] Optionally, the calculation method of the test phase is structured re-parameterization;

[0025] The structural re-parameters are:

[0026]

[0027] Among them, F l represents the lth output feature, represents the convolution operation, W is the weight parameter of the convolution layer, μ is the cumulative average of the batch normalization layer BN, σ is the standard deviation of the batch normalization layer BN, γ is the learning scaling factor of the batch normalization layer BN, β is the deviation of the batch normalization layer BN, W′, b′ are the weight and bias term of the convolution network after the structured heavy parameter transformation, γ i is the standard deviation in batch normalization of the i-th layer, σ i is the learned scaling factor in batch normalization of layer i, β i is the bias in batch normalization of the i-th layer.

[0028] Optionally, the double decoupled network model is decoupled in the training phase and the testing phase respectively, including:

[0029] The double decoupled network model is decoupled in the training phase including a first phase and a second phase;

[0030] The first stage is the feature learning stage, which extracts data abstract features through a multi-branch topological structure;

[0031] The second stage is classifier learning, which is used for output classification during training;

[0032] The decoupling of the double decoupling network model in the testing phase includes: when the training accuracy of the double decoupling network model meets the threshold condition, the convolutional network with a multi-branch topological structure is converted into a single-branch structure through the structured re-parameterization.

[0033] Optionally, inputting the training set into the dual decoupled network model, and implementing long-tail activity recognition through the mobile wearable recognition device includes:

[0034] Inputting the training set into the double decoupling network model for training, determining the training accuracy of the double decoupling network model, and obtaining the trained double decoupling network model parameters;

[0035] When the training accuracy of the double decoupling network model meets the threshold condition, the training is terminated, the parameters of the double decoupling network model after the training are obtained, and the parameters are saved in the form of a multidimensional array;

[0036] When the training accuracy of the double decoupling network model does not meet the threshold condition, adjusting the hyperparameters of the double decoupling network model on the mobile terminal;

[0037] The trained dual decoupled network model parameters are input into the mobile wearable recognition device, and data is loaded through the embedded sensor to realize long-tail activity recognition.

[0038] Optionally, the mobile wearable identification device includes a mobile wearable identification device having a neural network processing unit.

[0039] The present invention has the following beneficial effects:

[0040] (1) Compared with the activity recognition model based on conventional convolutional networks, the present invention enhances the feature extraction capability of the model by decoupling the training phase, that is, separating feature extraction and activity classification, and adopts a multi-branch topology structure. At the same time, the weight fine-tuning in the classifier phase can effectively improve the model's performance in recognizing long-tail distributions.

[0041] (2) In the present invention, the decoupling of the training phase and the testing phase significantly reduces the number of model parameters, which helps to reduce the hardware requirements of wearable recognition devices and improve the real-time performance of activity recognition;

[0042] (3) The mobile-side double-decoupled network model of the present invention is not only applicable to data sets with balanced distribution of activity data samples, but also to more common long-tail activity data sets. It has a wide range of applications, strong robustness, and can be seamlessly transplanted to smart wearable devices, showing potential for commercial applications. In addition, the method of the present invention is applicable to the situation where human activity data collected by sensors usually follows a long-tail distribution, and can achieve high-accuracy category recognition without sacrificing the recognition performance for long-tail categories.

[0043] (4) The method of the present invention does not require additional computational burden and can be applied to mobile devices such as smart phones and watches, and has high practicality and applicability;

[0044] (5) The present invention has been experimentally verified on multiple public datasets and has achieved the best performance in the current field of sensor long-tail activity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 This is a flow chart of a method for identifying long-tail activities in a dual-decoupled network on a mobile terminal according to an embodiment of the present invention;

[0047] Figure 2A schematic diagram of a dual decoupling network on a mobile terminal proposed in an embodiment of the present invention;

[0048] Figure 3 The following are visualization diagrams of the weight norm evolution process of different model networks proposed in the embodiments of the present invention, wherein (a) is a visualization diagram of the weight norm evolution process of a common convolutional network, (b) is a visualization diagram of the weight norm evolution process of a multi-branch convolutional network, and (c) is a visualization diagram of the weight norm evolution process of a mobile-side double-decoupled convolutional network;

[0049] Figure 4 A comparison diagram of the classifier weight distribution and its evolution proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0051] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] The present invention proposes a novel dual-stage wise decoupling (DSWD) method. Aiming at the problem of long-tail class distribution in sensor activity recognition, the method is based on class balance loss and structured reparameterization technology, and the performance of sensor activity recognition is improved by decoupling the training process and the reasoning structure. The DSWD method adopts a staged activity recognition training strategy, and the training stage is decoupled into two parts: first, the backbone network is trained to learn the feature representation of long-tail activities; then, the classification network is fine-tuned in combination with class balance loss, weight decay and MaxNorm. The reasoning structure decoupling is to decouple the multi-branch topological convolutional neural network of the backbone network into a single-branch network, thereby reducing the amount of model parameters, reducing the hardware threshold of wearable recognition devices, and enhancing real-time recognition capabilities. The DSWD method has good versatility and can be applied to various sensor-based activity recognition tasks. The advantages of the present invention are that the performance of sensor activity recognition is improved, the amount of model parameters is reduced, the hardware threshold of wearable recognition devices is reduced, and the real-time recognition capability is enhanced, and at the same time, it has practical application value and market prospects.

[0053] like Figure 1As shown, a method for long-tail activity recognition using a mobile dual-decoupled network is provided in this embodiment. The method first collects time series signals of human activities from wearable sensors, and after noise reduction and normalization processing, inputs them into the mobile dual-decoupled network of the present invention for model training. After training, the model network is transplanted to a wearable device with an NPU function, and finally the recognition of long-tail activities is realized. This method adopts a dual-decoupled network structure and a phased training strategy, combined with existing technologies such as class balance loss, structured reparameterization technology, weight decay and MaxNorm, to improve the accuracy and generalization ability of long-tail activity recognition. Specifically, it includes the following steps:

[0054] Step 1: Collect action signals;

[0055] Determine the long-tail activity types. The imbalance of the sample size of each activity type should not be higher than 13 (sample size of the most types / sample size of the least types). Collect the required action signals through a mobile wearable recognition device with embedded sensors, and mark the corresponding action types according to the timestamp of the synchronously recorded collection video.

[0056] Step 2: Perform data preprocessing and data division on the collected motion signals to obtain the final data set;

[0057] Among them, preprocessing of action signals includes filtering, noise reduction, and normalization preprocessing operations; data division of action signals includes: setting time windows, sliding windows, and data overlap rates for action signals according to the sampling frequency of the acquisition sensor device, and dividing the preprocessed action data to form a data set including a verification set, a training set, and a test set.

[0058] Step 3: Train the mobile-side double decoupling network model based on the final data set to obtain the trained mobile-side double decoupling network model parameters;

[0059] like Figure 2As shown, the structure of the figure includes the double decoupling process of DSWD in this embodiment. Among them, the decoupling of the first stage is the decoupling of the training stage, which is divided into two stages: the multi-branch feature learning stage and the classifier fine-tuning stage. The multi-branch feature learning stage realizes the use of cross entropy loss and weight decay, as well as the feature learning of the multi-branch backbone network, that is, the first stage is the feature learning stage, which extracts data abstract features through the multi-branch topological structure; the classifier fine-tuning stage is realized by jointly adjusting the class balance loss, weight decay and MaxNorm and other technologies, that is, the second stage is classifier learning, which is used for output classification during training. The decoupling of the second stage is the decoupling of the testing stage, which is the decoupling of the multi-branch topological structure to the single-branch topological structure. The zero filling of the convolution layer is mainly realized by the re-parameterization technology. Specifically, when the training accuracy meets the conditions, the training is terminated and the model parameters are saved. Finally, the convolutional network with a multi-branch topological structure is converted into a single-branch structure through structured re-parameterization.

[0060] The mobile dual decoupled network model includes a training phase and a testing phase:

[0061] Training phase: three layers of convolution blocks with a dual-branch topology and one fully connected classification output layer. Each convolution block contains a convolution layer, a batch normalization layer BN, and a ReLU activation function. The three layers of convolution blocks are used to extract feature information of activity data, and the fully connected classification layer is used to output classification.

[0062] Testing phase: three layers of single-branch convolution blocks and one fully connected classification output layer, where each convolution block contains a convolution layer, a batch normalization layer BN, and a ReLU activation function. The three layers of single-branch convolution blocks are only used for forward reasoning, and the fully connected classification layer is used for output classification.

[0063] The calculation method of the training phase based on the mobile dual decoupled network model mainly includes the minimization and fine-tuning of three weight parameters, namely, minimization of the class balance error function, fine-tuning of the L2 penalty term, and fine-tuning of the MaxNorm restriction term.

[0064] The class balance error function is minimized as follows:

[0065] W * =argminCB(f(x i ;w),y i )

[0066] Among them, W * is the overall weight parameter of the fully connected layer, argmin is the function to obtain the minimum value, CB represents the class balance error function, f is the mapping function of the fully connected layer, x i is the activity data matrix for each batch, w is the corresponding x i The weight parameter, y i is the true label.

[0067] The functional form of L2 penalty fine-tuning is as follows:

[0068]

[0069] Among them, k is a hyperparameter that controls the degree of decay.

[0070] The function form of MaxNorm limit fine-tuning is as follows:

[0071]

[0072] Among them, st means that the two norm of the weight w should be less than the threshold r 2 value.

[0073] The calculation method of the test phase based on the mobile terminal double decoupled network model is structured heavy parameterization, and its function form is as follows:

[0074]

[0075] Among them, F l represents the lth output feature, represents the convolution operation, W is the weight parameter of the convolution layer, μ is the cumulative average of the batch normalization layer BN, σ is the standard deviation of the batch normalization layer BN, γ is the learning scaling factor of the batch normalization layer BN, β is the deviation of the batch normalization layer BN, W′, b′ are the weight and bias term of the convolution network after the structured heavy parameter transformation, γ i is the standard deviation in batch normalization of the i-th layer, σ i is the learned scaling factor in batch normalization of layer i, β i is the bias in batch normalization of the i-th layer.

[0076] Training based on the mobile dual decoupled network model includes:

[0077] The pre-trained activity data is sent to the constructed mobile dual-decoupled network for training. When the mobile dual-decoupled network model meets the training accuracy requirements (the minimum accuracy requirement is not less than 85%), the training is terminated, and the model parameter weights, bias and other parameter information, that is, the long-tail recognition results are saved in the computer disk in the form of a multidimensional array; conversely, if the recognition accuracy does not meet the recognition requirements, the hyperparameters of the mobile dual-decoupled network model are adjusted.

[0078] Among them, the recognition accuracy is a percentage, the higher the percentage, the stronger the recognition effect, and it should be greater than 85%. The recognition accuracy can be adjusted according to different recognition requirements, and the mobile wearable recognition device includes a mobile wearable recognition device with a neural network processing unit.

[0079] Step 4: Import the trained mobile dual-decoupled network model parameters into the mobile wearable device, and load the activity data through the embedded sensor to achieve the purpose of long-tail activity recognition.

[0080] like Figure 3 As shown in , during the training process, the weights of the common network and the multi-branch CNN grow faster, and the weights in the main categories gradually increase. These models achieve higher recognition accuracy by increasing the classifier weights of the head categories, for example, Figure 3 As shown in (a), the color bar of the ordinary convolutional network is mainly concentrated on the action type with the largest number of samples, which reflects that it focuses on the category with a large number and uses this strategy to improve the overall recognition accuracy. Figure 3 As shown in (b), there are significant differences between the color bars of each action, indicating that the head category has a larger weight and the tail category has a smaller weight. Figure 3 As shown in (c), the proposed model has similar weight evolution for each activity, and the color bar for each category is uniform and balanced, indicating the robustness of the proposed method on the LTR task. The proposed model promotes balanced weight learning by lowering the weight curve and adjusting the weights of rare categories. Good results are shown by combining MaxNorm, weight decay, and class balance loss during training.

[0081] like Figure 4 The figure shows the comparison of classifier weight distribution of different techniques in terms of balanced weights in the first stage of training. Consistent with the hypothesis, the native method pays more attention to the head categories, such as "walking" and "going upstairs", without emphasizing the accuracy of rare categories. L2 normalization makes the classifier weights evenly distributed. However, this uniform weight distribution does not improve the overall recognition accuracy. Empirically, this approximately linear weight distribution is suitable for balanced datasets. For long-tail datasets, the weight distribution shows that the weight value of "head categories" is larger than that of "tail categories", but the gap is small. Weight decay further narrows the gap between the majority and minority categories, slightly alleviating the norm imbalance compared to the native method. In contrast, MaxNorm promotes balanced weight learning by lowering the weight curve and adjusting the weights of rare categories. During the training process, the proposed model shows good results by combining MaxNorm, weight decay and CB loss. The figure shows the evolution of the classifier weight norm of each type of network during training. The x-axis represents the training cycle, and the y-axis represents the six activity categories in UCI-HAR_LT-100 (the imbalance degree λ of UCI-HAR is 100) (the number of samples increases from bottom to top).

[0082] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A mobile terminal dual decoupled network long-tail activity identification method, characterized in that: The specific steps include: Collect action signals and obtain training sets; Collecting the action signal and obtaining the training set includes: The motion signal is collected by a mobile wearable recognition device, the motion signal is preprocessed and data is divided to obtain the training set, and the corresponding motion type is marked according to the timestamp of the synchronously recorded acquisition video; Inputting the training set into a double decoupled network model, and realizing long-tail activity recognition through a mobile wearable recognition device; Wherein, the double decoupled network model is constructed by a neural network and is decoupled in the training phase and the testing phase respectively; The double decoupled network model includes a convolution block with a three-layer topology structure and a fully connected classification output layer, wherein each of the convolution blocks includes a convolution layer and a batch normalization layer, and adopts a ReLU activation function, wherein the convolution block is used to extract feature information of activity data in the training phase and is used for forward reasoning in the testing phase; the fully connected classification output layer is used to output classification; The calculation method in the training phase includes minimization and fine-tuning of weight parameters, which are respectively minimization of class balance error function, fine-tuning of L2 penalty term, and fine-tuning of MaxNorm restriction term; The class balance error function minimization form is: W * =argminCB(f(x i ;w),y i ) The functional form of the L2 penalty term fine-tuning is: The function form of the MaxNorm limit item fine-tuning is: Among them, W * is the overall weight parameter of the fully connected layer, argmin is the function to obtain the minimum value, CB represents the class balance error function, f is the mapping function of the fully connected layer, x i is the activity data matrix for each batch, w is the corresponding x i The weight parameter, y i is the true label, k is a hyperparameter that controls the attenuation, and st means that the second norm of the weight w should be less than the threshold r 2 value; The calculation method of the test phase is structured re-parameterization; The structural re-parameters are: Among them, F l represents the lth output feature, represents the convolution operation, W is the weight parameter of the convolution layer, μ is the cumulative average of the batch normalization layer BN, σ is the standard deviation of the batch normalization layer BN, γ is the learning scaling factor of the batch normalization layer BN, β is the deviation of the batch normalization layer BN, W', b' are the weight and bias term of the convolution network after the structured heavy parameter transformation, γ i is the standard deviation in batch normalization of layer i, σ i is the learned scaling factor in batch normalization of layer i, β i is the bias in batch normalization of the i-th layer; The double decoupled network model is decoupled in the training phase and the testing phase respectively, including: The double decoupled network model performs decoupling in the training phase, including the first phase and the second phase; the multi-branch feature learning phase realizes feature learning using cross entropy loss and weight decay, as well as multi-branch backbone network, that is, the first phase is the feature learning phase, extracting data abstract features through a multi-branch topological structure; The classifier fine-tuning stage is achieved by adjusting class balance loss, weight decay and MaxNorm together. That is, the second stage is classifier learning, which is used for output classification during training; The decoupling of the double decoupling network model in the testing phase includes: when the training accuracy of the double decoupling network model meets the threshold condition, converting the convolutional network with a multi-branch topological structure into a single-branch structure through the structured re-parameterization; Inputting the training set into the double decoupled network model, and realizing long-tail activity recognition through the mobile wearable recognition device includes: Inputting the training set into the double decoupling network model for training, determining the training accuracy of the double decoupling network model, and obtaining the trained double decoupling network model parameters; When the training accuracy of the double decoupling network model meets the threshold condition, the training is terminated, the parameters of the double decoupling network model after the training are obtained, and the parameters are saved in the form of a multidimensional array; When the training accuracy of the double decoupling network model does not meet the threshold condition, adjusting the hyperparameters of the double decoupling network model on the mobile terminal; Inputting the trained dual decoupled network model parameters into the mobile wearable recognition device for data loading to realize long-tail activity recognition; The mobile wearable identification device includes a mobile wearable identification device having a neural network processing unit.

2. The method for identifying long-tail activities in a mobile dual-decoupled network as claimed in claim 1, characterized in that: Preprocessing and data division of the action signal includes: Performing filtering, noise reduction and normalization processing on the motion signal to obtain a preprocessed motion signal; The sampling frequency of the acquisition sensor device; The time window, sliding window and data overlap rate of the action signal are set according to the sampling frequency, and the pre-processed action signal is divided into data.

Images