A time series data classification method based on the combination of global and local information

Abstract

The present invention discloses a time series data classification method based on the combination of global and local information. The method first selects the data with the richest information from each period by means of reinforcement learning and merges them into input samples. Then, a local information module and a global information module are established and added to the original residual block in the ResNet18 network, so as to realize the feature extraction of the same period of the time series data and the feature interaction of different periods, and finally complete the classification of the time series information. This method adopts the local information module, which can strengthen the temporal connection of the sample itself, and the global information module can strengthen the interaction of the time series data at different times. Compared with the general 3D convolutional network, it reduces the amount of calculation, improves the efficiency and accuracy of classification, and has certain reference value for data classification tasks with temporal relationships.

Description

Technical Field

[0001] The present invention belongs to the technical field of data classification, and relates to a method for classifying data containing time series relationships, and specifically to a time series data classification method based on the combination of global and local information. Background Art

[0002] Deep learning is based on a neural network system. Through multi-layer processing, it gradually transforms the initial "low-level" feature representation into a "high-level" feature representation, thereby learning the inherent laws and representation levels of the sample data. The model is then used to complete complex classification and other learning tasks, enabling machines to have analytical learning capabilities like humans and to recognize data such as text, images, and sounds.

[0003] Time series data refers to data that is indexed by the time dimension and has temporal connections. Time series data contains both local and global information. Local information refers to the connections within the time series itself, while global information refers to the connections between multiple segments of similar data. By studying the temporal changes in time series data, we can effectively capture its patterns. This can improve the accuracy of problem identification, reduce hardware costs, and increase the speed of problem detection. This provides valuable insights for other time series information-related application scenarios.

[0004] Common time series data includes multi-phase CT images and audio data. CT stands for computed tomography. Since cross-sectional images are scanned in a time series, CT images themselves also contain time series data. CT can be divided into plain CT and enhanced CT. Plain CT is conventional CT, while enhanced CT is images observed at different phases after injection of a contrast agent. Enhanced CT helps to observe details that are not visible with plain CT. In enhanced CT images, images from a single phase can be considered local information, while images from multiple phases can be considered global information. When audio data is acquired, its context includes time series relationships. Audio acquired from the same object over a continuous period of time can be considered local information, while multiple audio segments acquired intermittently can be considered global information.

[0005] When classifying time series data, existing technologies often ignore the feature information of individual samples across different time dimensions, effectively ignoring local information. Directly merging features across different samples also results in a low level of interaction between global information, leading to insufficient classification accuracy. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention proposes a time series data classification method based on the combination of global and local information, taking into account the interaction and connection between local and global information, strengthening the temporal relationship of the data itself, and improving the calculation speed and classification accuracy.

[0007] A time series data classification method based on the combination of global and local information includes the following steps:

[0008] Step 1: Collect a continuous segment of time series data for a sampling object and then split it into segments of equal length as a sample. Alternatively, collect multiple segments of time series data of equal length for a sampling object at intervals, each of which consists of time series data from multiple stages. Collect samples from multiple sampling objects and use the sample categories as labels to form a dataset with a one-to-one correspondence between samples and labels.

[0009] The time series data is a video image, an audio file, an EEG data or an enhanced CT image.

[0010] Step 2: Adjust the sample data obtained in step 1 to a uniform size, and then perform random rotation or flipping data augmentation operations to obtain input samples.

[0011] As a preference, for the sample data obtained in step 1, first adjust it to a uniform size to obtain X img ∈R Q ×H×W×C×M , where Q represents the amount of data of a sample in one stage, HxW represents the sample size, C represents the number of sample channels, and M represents the number of stages of a sample. Then, using the reinforcement learning method, we select t data with the richest information in each period to form the input sample X∈R N×T×H×W×C , where N represents the batch size of the input samples and T represents the amount of data of the input samples.

[0012] Step 3: Use the local information module to stimulate the local features of the input sample. The specific steps are as follows:

[0013] s3.1, use 1×1 2D convolution kernel to reduce feature channels to improve efficiency, and obtain the feature X after channel reduction r :

[0014] X r =conv red *X ri (1)

[0015] in, represents the characteristics after channel reduction, represents the input features of the local information module, C ri Represents the channel of the input feature, H ri and W ri Represents the length and width of the feature, conv red Represents a 2D convolution with a channel compression rate of red and a convolution kernel size of 1×1.

[0016] As a preference, set conv red The channel compression rate red=16.

[0017] s3.2, perform channel-level transformation on the features and use the transformed features to calculate the difference information X between different data L :

[0018] X L =(conv trans *X r (t+1)-X r (t))+(conv trans *X r (t)-X r (t-1))1≤t≤T-1 (2)

[0019] Among them, conv trans Represents a 2D convolution kernel with a size of 3×3,

[0020] s3.3, obtain spatial information X through the global average pooling layer Pool() Avg :

[0021] X Avg =Pool(X L ) (3)

[0022] in,

[0023] s3.4, using 2D convolution with a convolution kernel size of 1×1, the spatial information X Avg The channel dimension is expanded to the same channel dimension as the input feature, and the weight value A is obtained through the Sigmoid function:

[0024] A=Sigmoid(conv ex *X Avg ) (4)

[0025] in conv ex Represents a 2D convolution with a kernel size of 1×1.

[0026] s3.5, the input feature X of the local information module ri Multiply it by the weight value A obtained by the Sigmoid function to suppress useless information, and use residual links to enhance the three-dimensional information, and finally obtain the three-dimensional information feature X Local :

[0027] X Local =X ri +A⊙X ri (5)

[0028] in, ⊙ represents the matrix element-wise multiplication operation.

[0029] Step 4: Use the global information module to enhance the three-dimensional information feature X Local The specific steps for contact during different periods are as follows:

[0030] s4.1, according to the channel order, the three-dimensional information feature X Local Divided into eight characteristic parts, X Local =[X1,X2,…,X8], where

[0031] s4.2. Shift X1 and X3 backward in time, and shift X2 forward in time. Fill in any missing features in the current feature section, ensuring that each feature section contains feature information from both the upper and lower slices, and some feature sections also contain feature information from other times.

[0032] s4.3 Reorganize the features after the movement change to obtain the global information features

[0033] Step 5: Use the local information module and global information module described in steps 3 and 4, add them to the residual block of ResNet18, and establish a learning model for data processing. The specific steps are as follows:

[0034] s5.1、Input sample X∈R N×T×H×W×C Input the convolution kernel Conv1 with a size of 7×7, a stride of 2, an input channel of 1, and an output channel of 64, and obtain X1:

[0035] X1=Conv1*X (6)

[0036] in H1 and W1 represent the length and width after the first convolution kernel.

[0037] s5.2, input X1 into the maximum pooling layer maxpool with a size of 3×3 and a step size of 2, and get X max :

[0038] X max =maxpool*X1 (7)

[0039] in H m and W m Represents the length and width after the maximum pooling layer.

[0040] s5.3, X maxThe network passes through four residual layers in sequence. Each residual layer includes the same residual block. Each residual block first extracts features through the local information module, the global information module and two convolution modules with a convolution kernel size of 3x3. The extracted data is then added to the input data. The number of output channels of the four residual layers is 64, 128, 256 and 512 respectively. Finally, the output of the fourth residual layer is obtained. H6 and W6 represent the length and width of the feature data after passing through the fourth residual layer.

[0041] s5.4, X Res_4 Input the average pooling layer and the fully connected layer in sequence to get the classification result Prob out :

[0042] Prob out =FC(Pool(X Res_4 )) (8)

[0043] Among them Prob out ∈R N×k , k represents the label type, and FC() represents the fully connected layer.

[0044] Step 6: Classify the data set samples obtained in step 1 through step 5, input the classification results into the Softmax layer, and obtain the classification label Compared with the true label y of the sample, calculate the cross entropy loss Loss:

[0045]

[0046] Then the stochastic gradient descent algorithm is used to optimize the learning model in step 5.

[0047] Step 7: Use the learning model optimized in step 6 as a time series data classification model, collect time series data with unknown categories, input it into the time series classification model after preprocessing, obtain the corresponding category label, and complete the classification task.

[0048] The present invention has the following beneficial effects:

[0049] This method uses local information modules to strengthen the feature connections between time series data within a certain period of time, and global information modules to strengthen the interactive relationships between time series data in different time periods. Based on ResNet18, the structure of the residual block is modified. This speeds up computation compared to conventional 3D networks, reduces excessive training losses, maintains the model's structural stability, reduces computational redundancy and latency, and improves classification accuracy while increasing classification speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1It is a flow chart of a temporal information classification method based on the combination of global and local information;

[0051] Figure 2 is a schematic diagram of a local information module in an embodiment;

[0052] Figure 3 is a schematic diagram of a global information module in an embodiment;

[0053] Figure 4 Schematic diagram of a learning model established in the embodiment;

[0054] Figure 5 is a classification flow chart for audio timing information in an embodiment; DETAILED DESCRIPTION

[0055] The present invention will be further explained below with reference to the accompanying drawings;

[0056] Example 1

[0057] This embodiment takes liver cancer CT images as an example to illustrate how to classify different liver cancer images using a time series data classification method based on the combination of global and local information. Figure 1 As shown, the specific steps include:

[0058] Step 1: Collect plain CT images of liver cancer from different patients and enhanced CT images from the arterial, portal venous, and delayed phases after contrast agent injection. Images from one patient at four different phases are used as a sample, and the cancer types in the images are annotated, including hepatocellular carcinoma (HCC) and intrahepatic cholangiocarcinoma (ICC). Collect 200 HCC images and 200 ICC images, with the HCC image label set to 0 and the ICC image label set to 1. Randomly select 320 of these samples as the training set, and 80 samples as the test set.

[0059] Step 2: Preprocess the sample images in the training set, specifically including the following steps:

[0060] s2.1. First, use ITK-SNAP software to mark and segment the lesion part in the sample image.

[0061] s2.2. Due to individual differences in the subjects, the number of slices in the sampled CT images varies. In this example, the sample images are uniformly adjusted to a size of 64×128×128×4, where 64 represents the number of CT images in each phase, 128 and 128 represent the length and width of each CT image, and 4 indicates that one sample contains four phases.

[0062] s2.3. Using the reinforcement learning method, select a sample of 16 images with the richest information among the 64 CT images in each phase and retain them.

[0063] Reinforcement learning can describe and solve the problem of an intelligent agent learning strategies to maximize rewards or achieve specific goals during its interaction with the environment. Reinforcement learning transforms the problem of selecting CT slices into a sequential decision-making problem, enabling the design of optimal selections. Because differentiating between ICC and HCC is difficult, using reinforcement learning for selection allows the model to focus on more lesion details.

[0064] s2.4. Perform a data augmentation operation of random rotation or flipping on the image filtered by s2.3 to obtain an input sample.

[0065] Step 3: Figure 2 As shown in Figure 2, the local information module is used to stimulate the local features of the input sample. The specific steps are as follows:

[0066] s3.1, use 1×1 2D convolution kernel to reduce feature channels to improve efficiency, and obtain the feature X after channel reduction r :

[0067] X r =conv red *X ri (1)

[0068] Among them, X r ∈R 2×64×4×32×32 represents the feature after channel reduction, X ri ∈R 2×64×64×32×32 Represents the input features of the local information module, conv red It represents a 2D convolution with a channel compression rate of red=16 and a convolution kernel size of 1×1.

[0069] s3.2, perform channel-level transformation on the features and use the transformed features to calculate the difference information X between different slices L :

[0070] X L =(conv trans *X r (t+1)-X r (t))+(conv trans *X r (t)-X r (t-1))1≤t≤T-1 (2)

[0071] Among them, conv trans Represents a 2D convolution kernel with a size of 3×3, XL ∈R 2×64×4×32×32 .

[0072] s3.3, obtain spatial information X through the global average pooling layer Pool() Avg :

[0073] X Avg =Pool(X L ) (3)

[0074] Among them, X Avg ∈R 2×64×16×1×1 .

[0075] s3.4, using 2D convolution with a convolution kernel size of 1×1, the spatial information X Avg The channel dimension is expanded to the same channel dimension as the input feature, and the weight value A is obtained through the Sigmoid function:

[0076] A=Sigmoid(conv ex *X Avg ) (4)

[0077] where A∈R 2×64×64×1×1 , conv ex Represents a 2D convolution with a kernel size of 1×1.

[0078] s3.5, the input feature X of the local information module ri Multiply it by the weight value A obtained by the Sigmoid function to suppress useless information, and use residual links to enhance the three-dimensional information, and finally obtain the three-dimensional information feature X Local :

[0079] X Local =X ri +A⊙X ri (5)

[0080] Among them, X Local ∈R 2×64×64×32×32 , ⊙ represents the matrix element-wise multiplication operation.

[0081] Step 4: Figure 3 As shown, the global information module is used to enhance the three-dimensional information feature X Local The specific steps for contact during different periods are as follows:

[0082] s4.1, according to the channel order, the three-dimensional information feature X Local Divided into eight characteristic parts, X Local =[X1,X2,…,X8], where X i ∈X 2×64×8×32×32 , i=1,2,...8.

[0083] s4.2. Shift X1 and X3 backward in time, and shift X2 forward in time. Fill in any missing features in the current feature section, ensuring that each feature section contains feature information from both the upper and lower slices, and some feature sections also contain feature information from other times.

[0084] s4.3 Reorganize the features after the shift and obtain the global information feature X Global ∈R 2×64×64×32×32 .

[0085] Step 5: Use the local information module and global information module described in steps 3 and 4, add them to the residual block of ResNet18, and build a learning model for data processing, such as Figure 4 The specific steps are as follows:

[0086] s5.1、Input sample X∈R 2×64×128×128×1 Input the convolution kernel Conv1 with a size of 7×7, a stride of 2, an input channel of 1, and an output channel of 64, and obtain X1:

[0087] X1=Conv1*X (6)

[0088] where X1∈R 2×64×64×64×64 .

[0089] s5.2, input X1 into the maximum pooling layer maxpool with a size of 3×3 and a step size of 2, and get X max :

[0090] X max =maxpool*X1 (7)

[0091] where X max ∈R 2×64×64×32×32 .

[0092] s5.3, X max It passes through four residual layers in sequence. Each residual layer includes the same residual block. Each residual block first extracts features through the local information module, the global information module and two convolution modules with a convolution kernel size of 3x3. The extracted data is then added to the input data. The number of output channels of the difference layer is 64, 128, 256 and 512 respectively. Finally, X is obtained at the output of the fourth residual layer. Res_4 ∈R 2×64×512×4×4 .

[0093] s5.4, X Res_4 Input the average pooling layer and the fully connected layer in sequence to get the classification result Prob out :

[0094] Prob out =FC(Pool(X Res_4 )) (8)

[0095] Among them Prob out ∈R 2×2 , FC() represents the fully connected layer.

[0096] Step 6: Classify the training set through step 5, input the classification results into the Softmax layer, and obtain the classification label Compared with the true label y of the sample, calculate the cross entropy loss Loss:

[0097]

[0098] Then the stochastic gradient descent algorithm is used to optimize the learning model in step 5.

[0099] Step 7: Use the learning model optimized in step 6 as the time series data classification model, input the test set data, and input it into the time series classification model after preprocessing to obtain the corresponding category label to complete the classification task.

[0100] In this example, the traditional ResNet18 network was trained and tested using the same dataset. The classification accuracy for HCC and ICC was 77%, while the classification accuracy using this method increased to 83%, demonstrating that the local information module and global information module proposed in this method have a certain effect on improving the classification accuracy.

[0101] Example 2

[0102] This embodiment uses audio data as an example to illustrate how to achieve speech-based emotion classification through a time series data classification method based on the combination of global and local information. Figure 5 As shown, the specific steps include:

[0103] Step 1: Select a dataset from the AVEC2014 database. The audio data is collected using microphones under natural conditions. The subjects range in age from 18 to 63, with an average age of 31.5. The number of collections for each subject ranges from one to four, with the interval between each collection for the same subject being approximately two weeks. Each collected audio piece is between 6 seconds and 4 minutes and 8 seconds long, and includes two collection formats: NORTHWIND and FREEFORM.

[0104] (1) NORTHWIND: The subject is asked to read the specified article content and the voice data of the reading article is recorded through the microphone.

[0105] (2) FREEFORM: The staff communicates with the subject through the headset microphone. The subject answers the questions asked by the staff, and the microphone then records the voice data of the subject's reply.

[0106] We selected NORTHWIND and FREEFORM audio samples from 150 subjects in the AVEC2014 database. For each audio file, we removed long periods of silence and concatenated them into a single continuous audio file. We then used a Hamming window to segment the continuous audio file into non-overlapping 60-frame audio segments. Each frame contained 1024 data points, and the overlap between the previous and next frames was 1 / 2 of the frame length. With an audio sampling rate of 44100 Hz, the duration of a single audio segment was [(60 + 1) × 1024 / 2] / 44100 = 0.708 seconds. After segmentation, we obtained 7548 audio segments, of which 5100 were randomly selected as the training set and 2448 as the test set.

[0107] Step 2: Preprocess the audio clips in the training set, extract the Mel frequency cepstral coefficients and formant, energy, zero-crossing rate and other audio features, perform normalization, convolution and dimension conversion, and obtain the input data X∈R in matrix form. 2×16×224×224×1 .

[0108] Step 3: Use the same method as in Example 1 to build a learning model. After completing the training with the training set data, input the test set data, output the emotion label of the corresponding audio clip, and complete the classification task.

[0109] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the claims of the present invention. The scope of protection claimed in this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims. The obvious changes or modifications derived therefrom are still within the scope of protection created by the present invention.

Claims

1. A temporal information classification method based on the combination of global and local information, characterized by: The specific steps include: Step 1: Collect a continuous time series data for a sampling object, and then split it into pieces of equal length as a sample; collect samples from multiple sampling objects, use the sample category as a label, and form a data set with a one-to-one correspondence between samples and labels; the time series data can be video images, audio files, EEG data, or enhanced CT images; Step 2: Adjust the sample data obtained in step 1 to a uniform size, and then perform random rotation or flipping data augmentation operations to obtain input samples; Step 3: Use the local information module to stimulate the local features of the input sample. The specific steps are as follows: s3.1, use 1×1 2D convolution kernel to reduce feature channels to improve efficiency, and obtain the feature X after channel reduction r : X r =conv red *X ri (1) in, represents the characteristics after channel reduction, represents the input features of the local information module, N represents the batch size of the input sample, T represents the amount of data of the input sample, C ri Represents the channel of the input feature, H ri and W ri Represents the length and width of the feature, conv red Indicates a 2D convolution with a channel compression rate of red and a convolution kernel size of 1×1; s3.2, perform channel-level transformation on the features and use the transformed features to calculate the difference information X between different data L : X L =(conv trans *X r (t+1)-X r (t))+(conv trans *X r (t)-X r (t-1)) 1≤t≤T-1 (2) Among them, conv trans Represents a 2D convolution kernel with a size of 3×3, s3.3, obtain spatial information X through the global average pooling layer Pool() Avg : X Avg =Pool(X L ) (3) in, s3.4, using 2D convolution with a convolution kernel size of 1×1, the spatial information X Avg The channel dimension is expanded to the same channel dimension as the input feature, and the weight value A is obtained through the Sigmoid function: A=Sigmoid(conv ex *X Avg ) (4) in conv ex Represents a 2D convolution with a convolution kernel size of 1×1; s3.5, the input feature X of the local information module ri Multiply it by the weight value A obtained by the Sigmoid function to suppress useless information, and use residual links to enhance the three-dimensional information, and finally obtain the three-dimensional information feature X Local : X Local =X ri +A⊙X ri (5) in, ⊙ represents the matrix element dot multiplication operation; Step 4: Use the global information module to enhance the three-dimensional information feature X Local The specific steps for contact during different periods are as follows: s4.1, according to the channel order, the three-dimensional information feature X Local Divided into eight characteristic parts, X Local =[X1,X2,…,X8], where s4.

2. Move X1 and X3 backward in the time dimension, and move X2 forward in the time dimension; for the feature information that deviates from the original time dimension after the movement, fill it into the vacant position of the current feature part; s4.3 Reorganize the features after the movement change to obtain the global information features Step 5: Use the local information module and global information module described in steps 3 and 4, add them to the residual block of ResNet18, and establish a learning model for data processing. The specific steps are as follows: s5.1、Input sample X∈R N×T×H×W×C Input the convolution kernel Conv1 with a size of 7×7, a stride of 2, an input channel of 1, and an output channel of 64, and obtain X1: X1=Conv1*X (6) in H1 and W1 represent the length and width after the first convolution kernel; HxW represents the input sample size, and C represents the number of channels; s5.2, input X1 into the maximum pooling layer maxpool with a size of 3×3 and a step size of 2, and get X max : X max =maxpool*X1 (7) in H m and W m Represents the length and width after the maximum pooling layer; s5.3, X max It passes through four residual layers in sequence. Each residual layer includes the same residual block. Each residual block first extracts features through the local information module, the global information module and two convolution modules with a convolution kernel size of 3x3. The extracted data is then added to the input data. The number of output channels of the four residual layers is 64, 128, 256 and 512 respectively. Finally, the output of the fourth residual layer is obtained. H6 and W6 represent the length and width of the feature data after the fourth residual layer; s5.4, X Res_4 Input the average pooling layer and the fully connected layer in sequence to get the classification result Prob out : Prob out =FC(Pool(X Res_4 )) (8) Among them Prob out ∈R N×k , k represents the label type, FC() represents the fully connected layer; Step 6: Classify the data set samples obtained in step 1 through step 5, input the classification results into the Softmax layer, and obtain the classification label Compared with the true label y of the sample, calculate the cross entropy loss Loss: Then use the stochastic gradient descent algorithm to optimize the learning model in step 5; Step 7: Use the learning model optimized in step 6 as a time series data classification model, collect enhanced CT images or video images, and input them into the time series classification model after preprocessing to achieve image-based target classification; or collect audio files, and input them into the time series classification model after preprocessing to achieve voice-based emotion classification.

2. The time series information classification method based on the combination of global and local information as claimed in claim 1, characterized in that: Step 1 is to collect multiple segments of time series data with the same duration for a sampling object as a sample. Each sample includes time series data of multiple stages. Samples from multiple sampling objects are collected, and the categories of the samples are used as labels to form a data set with a one-to-one correspondence between samples and labels.

3. The time series information classification method based on the combination of global and local information according to claim 1 or 2, characterized in that: For the sample data obtained in step 1, first adjust it to a uniform size to obtain X img ∈R Q×H×W×C×M , where Q represents the amount of data of a sample in one stage, and M represents the number of stages of a sample; then, using the reinforcement learning method, we select T data with the richest information in each period to form the input sample X∈R N×T×H×W×C .

4. The time series information classification method based on the combination of global and local information according to claim 1, characterized in that: Setting up conv red The channel compression rate red=16.

Images