Time series early classification method, terminal device and storage medium

Abstract

The application discloses a time sequence early classification method, a terminal device and a storage medium, constructs a training set by using time sequence data of human body actions; trains a neural network by using the training set; inputs the training set into the trained neural network to obtain classification probabilities of all moments of all data of the training set, calculates a probability exit threshold value by using the classification probabilities; inputs observable data of a moment t into the trained neural network to obtain a classification probability Pt of the moment t, takes a maximum value of the Pt, and if the maximum value is greater than the probability exit threshold value, stops inputting the observable data, and takes a classification result of the moment t as a final classification result of the observable data of the moment t. The application can adapt to continuously increasing new data, extract more distinguishing features, and improve the accuracy of early classification of time data; the application can adapt to sample content and difficulty, extract more class-specific features, and improve the accuracy of early classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of time series data classification, and in particular to a time series early classification method, a terminal device and a storage medium. BACKGROUND

[0002] In recent years, with the development of intelligent wearable devices, time series data of human motion can be obtained everywhere for personal health monitoring, smart home control, etc., and time series classification tasks have attracted widespread attention. However, for some time-sensitive specific applications such as the fall of the elderly, it is hoped that the time series will be classified as soon as possible, in addition, early classification of human activities helps to minimize the response time of the system, thereby improving the user experience. Therefore, it is of great research significance to classify time series data as soon as possible and as accurately as possible.

[0003] Early classification continuously inputs data over time, and the length of the data is constantly changing, so the features at different times are quite different, and therefore it is difficult for the classifier to classify time series of any length.

[0004] Traditional early time series classification methods can be divided into prefix-based methods, shaplets-based methods, and posterior probability-based methods. However, these methods usually require a large amount of time to train multiple classifiers for time series data of different lengths, and require a large amount of expert experience to design manual features or set exit thresholds. Compared with traditional methods, deep learning-based methods can automatically extract more effective features in this big data era.

[0005] Current deep learning-based early time series classification methods can be divided into one-stage methods and two-stage methods. Two-stage methods usually train a classification model using the training set in the first stage, and then set a certain exit rule or a fixed exit threshold in the second stage, and obtain the classification result when the classification probability meets the exit condition. One-stage methods usually establish a classification subnetwork and an exit subnetwork at the same time, train them, and the classification subnetwork obtains the classification result, and the exit subnetwork is used to indicate whether to exit at this time.

[0006] Since the input data of early classification is constantly changing, deep learning-based methods usually use recurrent neural networks to adapt to data of constantly changing lengths. Recurrent neural networks have a forgetting defect due to their recursive structure, and cannot well classify long time series, and their local feature extraction capability is poor. Some methods combine convolutional neural networks to extract local features, unfortunately, the parameters of the conventional convolution kernel are fixed, and have a fixed and same feature matching template for any time and any sample, without fully considering the intra-class and inter-class differences. SUMMARY

[0007] The technical problem solved by the present application is to provide a time series early classification method, a terminal device and a storage medium, which fully considers the intra-class difference and the inter-class difference and improves the accuracy of human motion time data classification in view of the deficiencies of the prior art.

[0008] To solve the above technical problems, the technical scheme adopted by the present application is as follows: a time series early classification method, comprising the following steps:

[0009] S1, constructing a training set by using time series data of human motion;

[0010] S2, training a neural network by using the training set;

[0011] S3, inputting the training set into the trained neural network to obtain classification probabilities of all time points of all data of the training set, and calculating a probability exit threshold value by using the classification probabilities;

[0012] S4, inputting observable data at time point t into the trained neural network to obtain a classification probability Pt at time point t, taking the maximum value of Pt, if the maximum value is greater than the probability exit threshold value, stopping to continue inputting observable data, and taking the classification result at time point t as the final classification result; otherwise, increasing the value of t by 1, and repeating step S4 until the exit condition is met.

[0013] After obtaining the classification probability, the present application calculates the probability exit threshold value by using the classification probability, and further determines whether to exit the training according to the size relationship between the probability exit threshold value and the classification probability. The threshold value of the present application is not a fixed threshold value, but is calculated according to the classification probability, so it can adapt to the changing characteristics of the input data of early classification. In the present application, if the exit condition is not met, the data continues to be input into the classifier (trained neural network) for classification as time goes on until the exit condition is met, which fully considers the intra-class difference and the inter-class difference, and greatly improves the accuracy of early time series classification.

[0014] In the present application, the neural network comprises:

[0015] a plurality of cascaded first convolution modules for extracting features of input data to obtain first features;

[0016] a plurality of cascaded second convolution modules inputting the first features and used for extracting high-level features of the input data;

[0017] an average pooling layer inputting the high-level features and outputting fusion features corresponding to different time length sequences;

[0018] a linear layer inputting the fusion features and outputting prediction scores of different categories;

[0019] an exponential normalization layer, configured to normalize the prediction score, and output a classification probability.

[0020] In the application, the input data is input to the convolution block (first convolution module) to extract bottom features, the bottom features are then input to the dynamic convolution block (second convolution module) to extract time-adaptive high-level features, the high-level features are then input to the average pooling layer to obtain fusion features corresponding to different time length sequences, the fusion features are then input to the linear layer to obtain prediction scores of different categories, and finally the scores are input to the exponential normalization layer to obtain output probabilities.

[0021] In the application, the first convolution module is a bottom convolution module, configured to extract bottom features (first features or primary features), and the second convolution module is a dynamic convolution module, configured to extract high-level features.

[0022] In the application, the cascade refers to sequential connection, for example, the output of the first first convolution module is connected to the input of the second first convolution module, the output of the second first convolution module is connected to the input of the third first convolution module, and so on.

[0023] In the application, the first convolution module comprises a first hollow causal convolution layer, a first normalization layer, a second hollow causal convolution layer, a second normalization layer and a first linear activation unit connected in sequence, and the input features of the first convolution module are connected to the output features in a residual manner.

[0024] In the application, the second convolution module comprises a dynamic convolution layer, a third normalization layer and a second linear activation unit, wherein the dynamic convolution layer comprises a convolution kernel generation module, the convolution kernel generation module is configured to generate a convolution kernel corresponding to each time by using input data, and the input data is feature-extracted by using the convolution kernel.

[0025] The convolution parameters of the conventional convolution block are fixed and irrelevant to the content and length of the sample, which is not conducive to the extraction of features of early classification flow data, therefore, after the initial low-level features are extracted by the convolution block, the dynamic convolution module is designed in the application, so that the features extracted by the dynamic convolution module can adapt to the changes of data content and length, and the classification accuracy is further improved.

[0026] In the application, the convolution kernel generation module comprises a first convolution layer, a rectified linear unit, a batch normalization layer and a second convolution layer connected in sequence.

[0027] For the same sample, the conventional convolution kernel uses the same convolution kernel in the whole time period, and the features are not distinguishable, and with the increase of data amount, the information gain is limited, compared with the conventional convolution kernel, the time-adaptive convolution module designed in the application generates a convolution kernel specific to the increased data with the passage of time, and can extract more distinguishable features.

[0028] The implementation process of calculating the probability exit threshold value by using the classification probability comprises:

[0029] sorting the classification probability and removing the repeated items in the classification probability;

[0030] taking the median of adjacent classification probability to obtain a series of threshold value candidates;

[0031] selecting the threshold value candidate with the minimum cost as the probability exit threshold value.

[0032] Directly specifying the threshold value of the data set requires a large amount of expert experience, and the threshold value is not suitable for all data sets and has poor generalization performance. The cost formula method is used in the present application, and the threshold value suitable for different data sets can be automatically obtained, and the threshold value conforming to the actual demand can be obtained according to the demand (the value of alpha is adjusted, for example: when there is a higher demand for classification accuracy, the value of alpha needs to be increased, and when there is a higher demand for early exit, the value of alpha needs to be reduced). The method of calculating the threshold value by using the cost formula also has the advantage of interpretability.

[0033] The cost of the threshold value candidate is calculated by using the following formula: Cost β =α*(1-Acc β )+(1-α)·Earliness β ;Wherein, Acc β is the accuracy calculated when the maximum value of the classification probability is greater than the threshold value candidate beta; Earliness β is the early exit calculated when the maximum value of the classification probability is greater than the threshold value candidate beta, and alpha is a weight coefficient.

[0034] In the present application, through a large number of experimental research and analysis, the value of alpha is 0.8.

[0035] As an inventive concept, the present application also provides a terminal device comprising a memory, a processor and a computer program stored in the memory; the processor executes the computer program to realize the steps of the above-mentioned method of the present application.

[0036] A computer readable storage medium having a computer program / instruction stored thereon; the computer program / instruction is executed by a processor to realize the steps of the above-mentioned method of the present application.

[0037] Compared with the prior art, the present application has the beneficial effects that:

[0038] 1. For data with increasing length, the present application can adapt to the increasing new data, extract more distinctive features, and improve the accuracy of early classification of time data;

[0039] 2. For different categories of data, this invention can adapt to the sample content and difficulty to extract more class-specific features, thereby improving the accuracy of early classification. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0041] Figure 2 This is a diagram of the neural network structure of Embodiment 1 of the present invention;

[0042] Figure 3 This is a structural diagram of the dynamic convolution module in Embodiment 1 of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] In this document, the terms "first," "second," and other similar words are not intended to imply any order, quantity, or importance, but are merely used to distinguish different elements. The terms "one," "a," and other similar words are not intended to indicate the existence of only one of the stated things, but rather that the description pertains to only one of the two stated things, which may include one or more. The terms "comprising," "including," and other similar words are intended to indicate a logical relationship, not a spatial relationship. For example, "A includes B" means that logically B belongs to A, not that spatially B is located inside A. Furthermore, the meanings of the terms "comprising," "including," and other similar words should be considered open-ended, not closed. For example, "A includes B" means that B belongs to A, but B does not necessarily constitute all of A; A may also include other elements such as C, D, and E.

[0045] Example 1

[0046] like Figure 1 As shown, in this embodiment, during the training phase, the network designed in this invention (DTCN, the specific structure of which is described in the next subsection) is first trained. Specifically, all time-series data from the training set are input into the network designed in this invention, and a loss function is used. Train all parameters of the training model, where N is the number of samples in the training set, and T is the length of the complete time series data. Then, the training set is input into the trained DTCN to obtain the classification probability of all data at all times. According to the exit rule formulated in the present application, the probability exit threshold of the data set is calculated. The specific process of calculating the threshold value by the exit rule is as follows: first, sort the classification probabilities and remove the repeated items to obtain {P1, P2, …, P T}. Then, the adjacent classification probabilities are taken as the median, such as β1=(P1+P2) / 2, to obtain a series of threshold value candidates {β1, β2, …, β T}. For each candidate threshold β, the present application defines a cost formula:

[0047] Cost β = α*(1-Acc β )+(1-α)*Earliness β .

[0048] Where Acc refers to the accuracy of the prediction probability P of the training set when greater than the threshold value, and Earliness refers to the early nature of the prediction probability P of the training set when greater than the threshold value. After calculating the cost of all candidate threshold values, the threshold value with the minimum cost is selected as the final exit threshold of the data. In this embodiment, the value of α is selected as 0.8.

[0049] In the test phase, taking sample A as an example, the observable data at time t (the observable data refers to the data of length t that can be obtained by equipment at time t, and the data of length t+1 can be obtained at t+1. The observable data at different times is slowly increasing over time) is directly input into the trained network model to obtain the classification probability Pt. Take the maximum value of Pt. If it is greater than the exit threshold value calculated in the training phase, it is considered that the classification result at this time is reliable, and the exit is performed. The classification result at this time is taken as the final classification result of the sample A. Otherwise, with the passage of time, the data continues to be input into the classifier for classification until the exit condition is met.

[0050] The architecture of the DTCN proposed in this embodiment is shown in Figure 2 . It is mainly composed of convolution blocks and dynamic convolution blocks. Specifically, the input data is input into the convolution block to extract the bottom layer features. Then, the bottom layer features are input into the dynamic convolution block to extract the time-adaptive high layer features. Then, the high layer features are input into the average pooling layer to obtain the fusion features corresponding to different time length sequences. These fusion features pass through the linear layer to obtain the prediction scores of different categories. Finally, the scores pass through the exponential normalization layer to obtain the output probability.

[0051] The bottom convolution block (first convolution module) is a convolution module for extracting bottom primary features (first features), and the convolution module comprises a dilated causal convolution, a normalization layer, and a rectified linear unit (ReLU). The reason for introducing the causal convolution is to avoid information leakage, so that the features at this moment are irrelevant to the features after this moment; the dilated convolution effectively expands the receptive field; the normalization layer avoids overfitting in the training process; and the ReLU enhances the nonlinearity of the extracted features. The convolution parameters of the conventional convolution block are fixed and irrelevant to the content and length of the sample, which is not conducive to the extraction of features of early classification flow data. Therefore, the dynamic convolution module is designed after the convolution block extracts the initial low-level features, so that the features extracted by the dynamic convolution module can adapt to the changes in data content and length.

[0052] The design of the dynamic convolution module in the embodiment is specifically as shown in Figure 3 The input features are generated into each time-specific convolution kernel through a convolution kernel generation module. In the training process, the convolution kernel generation module learns how to produce a convolution kernel that is adaptive to the data content. The convolution kernel generation module is composed of a convolution with a size of 1, a batch normalization layer, and a ReLU. The convolution kernel generation module produces a time-adaptive convolution kernel with a size of K and shared channels. The input features are convolved using the newly generated time-adaptive convolution kernel to obtain output features. For different categories of samples, the adaptive content convolution kernel is generated through the design of the dynamic convolution generation module, which can extract class-specific features; for the same sample, the conventional convolution kernel uses the same convolution kernel in the entire time period, and the features are not distinctive. With the increase of data, the information gain is limited. Compared with the conventional convolution kernel, the designed time-adaptive convolution module generates a convolution kernel specific to the increasing data over time, which can extract more distinctive features.

[0053] The time-adaptive features output by the dynamic convolution module are fused for each time period through an average pooling layer to obtain fused features of different time periods. Finally, the fused features at different time points pass through a linear layer and a normalization exponential layer to obtain the classification probability at each time point.

[0054] Experiments were conducted on two commonly used human motion recognition data sets, and ideal results were obtained (the index HM used in the table below is calculated by early and accuracy, and is a comprehensive index. The method of the embodiment of the application is DETSCN).

[0055] Table 1 Comparison of classification results

[0056]

[0057] In Table 1 above, the ECLN, ETMD, and EARLIEST methods are described as follows:

[0058] ECLN: Ruβwurm M, Tavenard R, Lefèvre S, et al. Early classification for agricultural monitoring from satellite time series [J]. arXiv preprint arXiv: 1908.10283, 2019.

[0059] ETMD: Sharma A, Singh S K, Udmale S S, et al. Early Transportation Mode Detection Using Smartphone Sensing Data [J]. IEEE Sensors Journal, 2021, 21(14): 15651-15659.

[0060] EARLIEST: Hartvigsen T, Sen C, Kong X, et al. Adaptive-halting policy network for early classification [C] / / Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019: 101-1.

[0061] Embodiment 2

[0062] Embodiment 2 of the present application provides a terminal device corresponding to the above-mentioned embodiment 1, which can be a processing device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0063] The terminal device of the present embodiment includes a memory, a processor and a computer program stored on the memory; the processor executes the computer program on the memory to realize the steps of the method of embodiment 1.

[0064] In some implementations, the memory can be a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory.

[0065] In other implementations, the processor can be a central processing unit (CPU), a digital signal processor (DSP) or various types of general-purpose processors, without limitation.

[0066] Embodiment 3

[0067] Embodiment 3 of the present application provides a computer readable storage medium corresponding to the above-mentioned embodiment 1, which has stored thereon computer programs / instructions. The computer programs / instructions are executed by a processor to implement the steps of the method of embodiment 1.

[0068] The computer readable storage medium can be a tangible device that maintains and stores instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0069] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0070] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable data processing apparatus to produce a computer-implemented process such that the instructions executed by the computer or other programmable data processing apparatus provide the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0072] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.

[0073] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A time series early classification method, characterized in that, The method comprises the following steps: S1, constructing a training set by using time series data of human actions; S2, training a neural network by using the training set; S3, inputting the training set into the trained neural network to obtain classification probabilities of all time points of all data in the training set, and calculating a probability exit threshold value by using the classification probabilities; S4, inputting observable data at time point t into the trained neural network to obtain a classification probability Pt at time point t, taking the maximum value of Pt, and if the maximum value is greater than the probability exit threshold value, stopping the input of observable data and taking the classification result at time point t as the final classification result of the observable data at time point t; Otherwise, increasing the value of t by 1 and repeating step S4; The neural network comprises: a plurality of cascaded first convolution modules for extracting features of input data to obtain first features; a plurality of cascaded second convolution modules for extracting high-level features of the input data, with the first features as input; an average pooling layer for inputting the high-level features and outputting fusion features corresponding to different time length sequences; a linear layer for inputting the fusion features and outputting prediction scores of different categories; an exponential normalization layer for normalizing the prediction scores and outputting classification probabilities; The first convolution module comprises a first dilated causal convolution layer, a first normalization layer, a second dilated causal convolution layer, a second normalization layer and a first linear activation unit connected in sequence; the input features of the first convolution module are connected in a residual manner with the output features. The second convolution module comprises a dynamic convolution layer, a third normalization layer and a second linear activation unit; the dynamic convolution layer comprises a convolution kernel generation module for generating a convolution kernel corresponding to each time point by using input data and extracting features of the input data by using the convolution kernel. The implementation process of calculating the probability exit threshold value by using the classification probabilities comprises: sorting the classification probabilities and removing repeated items in the classification probabilities; taking the median of adjacent classification probabilities to obtain a series of threshold candidate values; selecting a threshold candidate value with the minimum cost as the probability exit threshold value.

2. The time series early classification method of claim 1, wherein, The convolution kernel generation module comprises a first convolution layer, a rectified linear unit, a batch normalization layer and a second convolution layer connected in sequence.

3. The time series early classification method of claim 1, wherein, The cost of the threshold candidate value is calculated using the following formula: Cost β = a * (1 - Acc β ) + (1 - a) * Earliness β ; where Acc β is the accuracy calculated when the maximum of the classification probabilities exits greater than the threshold candidate value β; Earliness β is the earliness calculated when the maximum of the classification probabilities exits greater than the threshold candidate value β; and a is a weighting factor.

4. The time series early classification method of claim 3, wherein, The value of α is 0.

8.

5. A terminal device comprising a memory, a processor, and a computer program stored on the memory; characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1-4.

6. A computer readable storage medium having stored thereon computer programs / instructions; characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method in any one of claims 1-4.

Images