Multi-source time series signal classification method and device based on multi-scale feature fusion

Abstract

The application discloses a multi-source time sequence signal classification method and equipment based on multi-scale feature fusion. The method comprises the following steps: collecting multi-source sensor time sequence signals, and constructing a data set; constructing a multi-source time sequence signal classification model; training the multi-source time sequence signal classification model by using the data set to obtain a trained multi-source time sequence signal classification model; and classifying input multi-source sensor time sequence signals by using the trained multi-source time sequence signal classification model. The application effectively fuses multi-source sensor signals by using a deep learning method, obtains more robust feature representation through various fusion strategies, and more accurately and efficiently classifies multi-source time sequence signals.

Description

Technical Field

[0001] This invention relates to the field of time series signal classification technology, and in particular to a method and device for classifying multi-source time series signals based on multi-scale feature fusion. Background Technology

[0002] Currently, multi-source time series signal classification is widely used in industries, finance, and medicine, where deep neural network models play a crucial role. However, when faced with heterogeneous, multi-source time series signals, different signals may have certain correlations and contain different classification information, which limits the classification ability of deep neural network models in these fields and makes it difficult to achieve high classification accuracy (Classification methods, devices, equipment and storage media for multi-source time series (CN113920365A)).

[0003] In summary, how to effectively fuse multiple signals using deep learning, extract robust representations of the signals, and improve classification accuracy is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method and device for classifying multi-source time series signals based on multi-scale feature fusion, so as to achieve accurate and rapid classification of different time series signals.

[0005] The objective of this invention is achieved by at least one of the following technical solutions.

[0006] A multi-source time series signal classification method based on multi-scale feature fusion includes the following steps:

[0007] S1. Collect time-series signals from multiple sensor sources and construct a dataset;

[0008] S2. Construct a multi-source time series signal classification model that includes a sequentially connected signal preprocessing module, a multi-branch feature extractor, an encoder, a multi-scale feature fusion module, a cross-attention fusion module, and a first classifier;

[0009] S3. Use the dataset to train the multi-source time series signal classification model to obtain the trained multi-source time series signal classification model;

[0010] S4. Use the trained multi-source time series signal classification model to classify the input multi-source sensor time series signals.

[0011] Furthermore, in the multi-source time series signal classification model, the signal preprocessing module preprocesses the acquired multi-source sensor time series signals to remove noise signals generated during data acquisition, obtaining preprocessed multi-source sensor time series signals. The preprocessed multi-source sensor time series signals are input into a multi-branch feature extractor and an encoder to obtain multi-scale feature representations of the multi-source sensor time series signals. The multi-scale feature representations are input into a multi-scale feature fusion module to dynamically fuse feature representations of different scales of the multi-source sensor time series signals, obtaining a multi-scale fused feature vector of the multi-source sensor time series signals. The cross-attention fusion module fuses the multi-scale fused feature vectors of sensor time series signals from different sources to obtain a robust signal representation. The robust signal representation is input into the first classifier to obtain the classification result.

[0012] Furthermore, in the signal preprocessing module, the acquired multi-source sensor time series signals are preprocessed, including filtering, denoising, and smoothing, to remove noise signals generated during data acquisition.

[0013] Furthermore, in the multi-scale feature extractor, the preprocessed multi-source sensor time series signal is divided into time subsequences of different lengths on different scale branches. The time domain, frequency domain, time-frequency domain and location feature information of the corresponding time subsequences of different scale branches are extracted respectively, and then spliced ​​to generate the feature embedding of the corresponding time subsequence.

[0014] The features of multiple time subsequences of different lengths are embedded and input into an encoder of a set scale for feature encoding to obtain a multi-scale feature representation of the preprocessed multi-source sensor time series signal.

[0015] Furthermore, the multi-scale feature representation of the multi-source sensor time series signal is input into the multi-scale feature fusion module to dynamically fuse the different scale features of the multi-source sensor time series signal;

[0016] The multi-scale feature fusion module includes a second classifier. and information-sensing network The second classifier The output is the corresponding category y of M samples in the multi-source sensor time series signal. m probability distribution Among them, y m Let z be the corresponding category of the m-th sample. m For the b-th branch in the multi-branch feature extractor n The feature vector corresponding to the m-th sample in each branch, where m = 1 to M; B n b represents the number of branches in a multi-branch feature extractor. n =1~Bn ; It refers to the bth n In the branches, the second classifier The m-th sample output is of category y m The probability of.

[0017] Furthermore, after obtaining the class distribution probabilities of different samples in the multi-source sensor time series signal, the b-th element in the multi-scale feature extractor is defined. n The amount of feature information corresponding to each branch at each scale for:

[0018]

[0019] Among them, y m Let y represent the category corresponding to the m-th sample, y represent the set of labels for the M samples in a batch, and z represent the category of the b-th sample. n The set of features of M samples on each branch; · represents the inner product;

[0020] Information-sensing network Composed of a multilayer sensing mechanism, it is used to dynamically estimate the amount of information in different samples at different scales. Obtain information at different scales Then, the feature representation at that scale is obtained by dot product with the encoded feature vector z, and then the feature representations at different scales are concatenated to obtain the multi-scale fused feature vector.

[0021] Furthermore, after obtaining the multi-scale fusion feature vectors of multi-source sensor time series signals, in order to fully explore the correlation between sensor time series signals from different sources, the multi-scale fusion feature vectors of sensor time series signals from different sources are fused using the cross-attention fusion mechanism in the cross-attention fusion module to obtain robust signal representations.

[0022] Furthermore, in step S3, the multi-source time series signal classification model is trained using the dataset based on the total loss function L;

[0023] By maximizing the Kullback-Leibler divergence L between the predicted and true distributions cls For the second classifier Conduct training:

[0024]

[0025] in, Indicates the bth n The second classifier f of the branches bn The probability distribution of the m-th sample output;

[0026] In order to train the information-aware network Using L2 loss L info :

[0027]

[0028] The total loss function L = L cls +L info +L cel L cel This represents the overall classification loss of a multi-source time series signal classification model.

[0029] Furthermore, the overall classification loss L of the multi-source time series signal classification model cel The cross-entropy loss function is used.

[0030] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform steps of a multi-source time series signal classification method based on multi-scale feature fusion.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] This invention employs a deep neural network model that fuses time-series signals from multiple sensors to achieve end-to-end classification of these signals. By extracting and fusing features from the multi-source sensor signals, the model fully explores the correlations between different signals, resulting in high classification accuracy and sensitivity. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the steps of a multi-source time series signal classification method based on multi-scale feature fusion in an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the structure of the multi-source time series signal classification model in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the multi-scale feature fusion module in an example of the present invention. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] In order to use deep learning to achieve rapid and accurate screening of patients with dysphagia, the following issues need to be addressed: (1) How to extract robust representations of signals; (2) How to effectively fuse signals from different sources and explore the correlation between signals; (3) How the model can adaptively adjust to shield the effects of differences between different samples, as there may be differences between the sampled individuals and the sensors.

[0038] In one embodiment, a multi-source time series signal classification method based on multi-scale feature fusion, such as Figure 1 and Figure 2 As shown, it includes the following steps:

[0039] S1. Collect time-series signals from multiple sensor sources and construct a dataset;

[0040] In one embodiment, the collected data is labeled by professionals, and then the data is classified and organized to construct a dataset.

[0041] In one embodiment, the data acquired during the data acquisition process may be affected by external noise such as environmental noise.

[0042] S2, such as Figure 2 As shown, a multi-source time series signal classification model is constructed, comprising a signal preprocessing module, a multi-branch feature extractor, an encoder, a multi-scale feature fusion module, a cross-attention fusion module, and a first classifier connected in sequence.

[0043] In the multi-source time series signal classification model, the signal preprocessing module preprocesses the acquired multi-source sensor time series signals to remove noise signals generated during data acquisition, resulting in preprocessed multi-source sensor time series signals. The preprocessed multi-source sensor time series signals are then input into a multi-branch feature extractor and an encoder to obtain multi-scale feature representations of the multi-source sensor time series signals. These multi-scale feature representations are input into a multi-scale feature fusion module, which dynamically fuses feature representations from different scales of the multi-source sensor time series signals to obtain a multi-scale fused feature vector. A cross-attention fusion module fuses the multi-scale fused feature vectors of sensor time series signals from different sources to obtain a robust signal representation. This robust signal representation is then input into a first classifier to obtain the classification result.

[0044] In the signal preprocessing module, the acquired multi-source sensor time series signals are preprocessed, including filtering, denoising, and smoothing, to remove noise signals generated during data acquisition.

[0045] In the multi-scale feature extractor, the preprocessed multi-source sensor time series signal is divided into time subsequences of different lengths on different scale branches. The time domain, frequency domain, time-frequency domain and location feature information of the corresponding time subsequences of different scale branches are extracted respectively, and then spliced ​​to generate the feature embedding of the corresponding time subsequence.

[0046] The features of multiple time subsequences of different lengths are embedded and input into an encoder of a set scale for feature encoding to obtain a multi-scale feature representation of the preprocessed multi-source sensor time series signal.

[0047] The multi-scale feature representation of the time series signal from multiple sources is input into the multi-scale feature fusion module, which dynamically fuses the different scale features of the time series signal from multiple sources.

[0048] like Figure 3 As shown, the multi-scale feature fusion module includes a second classifier. and information-sensing network Where the second classifier f bn The output is the corresponding category y of M samples in the multi-source sensor time series signal. m probability distribution Among them, y m Let z be the corresponding category of the m-th sample. m For the b-th branch in the multi-branch feature extractor n The feature vector corresponding to the m-th sample in each branch, where m = 1 to M; B n b represents the number of branches in a multi-branch feature extractor. n =1~B n ; It refers to the bth n In the branches, the second classifier The m-th sample output is of category y m The probability of.

[0049] Furthermore, after obtaining the class distribution probabilities of different samples in the multi-source sensor time series signal, the b-th element in the multi-scale feature extractor is defined. n The amount of feature information corresponding to each branch at each scale for:

[0050]

[0051] Among them, y m Let y represent the category corresponding to the m-th sample, y represent the set of labels for the M samples in a batch, and z represent the category of the b-th sample. n The set of features of M samples on each branch; · represents the inner product;

[0052] Information-sensing network Composed of a multilayer sensing mechanism, it is used to dynamically estimate the amount of information in different samples at different scales. Obtain information at different scales Then, the feature representation at that scale is obtained by dot product with the encoded feature vector z, and then the feature representations at different scales are concatenated to obtain the multi-scale fused feature vector.

[0053] Furthermore, after obtaining the multi-scale fusion feature vectors of multi-source sensor time series signals, in order to fully explore the correlation between sensor time series signals from different sources, the multi-scale fusion feature vectors of sensor time series signals from different sources are fused using the cross-attention fusion mechanism in the cross-attention fusion module to obtain robust signal representations.

[0054] S3. Use the dataset to train the multi-source time series signal classification model to obtain the trained multi-source time series signal classification model;

[0055] The multi-source time series signal classification model is trained using the dataset based on the total loss function L;

[0056] By maximizing the Kullback-Leibler divergence L between the predicted and true distributions cls For the second classifier Conduct training:

[0057]

[0058] in, Indicates the bth n The second classifier of each branch The probability distribution of the m-th sample output;

[0059] In order to train the information-aware network Using L2 loss L info :

[0060]

[0061] The total loss function L = L cls +L info +L cel L cel This represents the overall classification loss of a multi-source time series signal classification model.

[0062] Furthermore, the overall classification loss L of the multi-source time series signal classification model cel The cross-entropy loss function is used.

[0063] S4. Use the trained multi-source time series signal classification model to classify the input multi-source sensor time series signals.

[0064] The preferred embodiments disclosed above are merely illustrative of the invention and its core concepts. Those skilled in the art will recognize that variations may occur in specific application scenarios and implementation based on the ideas of this invention; therefore, this specification should not be construed as limiting the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A classification method for multi-source time series signals based on multi-scale feature fusion, characterized in that, Includes the following steps: S1. Collect time-series signals from multiple sensor sources and construct a dataset; S2. Construct a multi-source time series signal classification model that includes a sequentially connected signal preprocessing module, a multi-branch feature extractor, an encoder, a multi-scale feature fusion module, a cross-attention fusion module, and a first classifier; In the multi-source time series signal classification model, the signal preprocessing module preprocesses the acquired multi-source sensor time series signals to remove noise signals generated during data acquisition, and obtains the preprocessed multi-source sensor time series signals; The preprocessed multi-source sensor time series signal is input into a multi-branch feature extractor and an encoder to obtain a multi-scale feature representation of the multi-source sensor time series signal; The multi-scale feature representation is input into the multi-scale feature fusion module, which dynamically fuses feature representations at different scales of time series signals from multiple sources to obtain a multi-scale fused feature vector of the time series signals from multiple sources; the cross-attention fusion module fuses the multi-scale fused feature vectors of time series signals from different sources to obtain a robust signal representation; the robust signal representation is input into the first classifier to obtain the classification result; The multi-scale feature representation of the time series signal from multiple sources is input into the multi-scale feature fusion module, which dynamically fuses the different scale features of the time series signal from multiple sources. The multi-scale feature fusion module includes a second classifier. and information-sensing network The second classifier The output is the time series signal from the multi-source sensor. The corresponding category of each sample probability distribution ,in, For the first The corresponding category of each sample, For the first branch feature extractor The first branch The feature vector corresponding to each sample ; This indicates the number of branches in the multi-branch feature extractor. ; It refers to the first In the branches, the second classifier The output of the first Each sample is a category The probability of; After obtaining the class distribution probabilities of different samples in the time series signal from multiple sensors, the first step in the multi-scale feature extractor is defined. The amount of feature information corresponding to each branch at each scale for: in, For the first The category corresponding to each sample Indicates a batch A set of labels for each sample. It indicates the first On each branch A set of features for each sample; It is the inner product; Information-sensing network Composed of a multilayer sensing mechanism, it is used to dynamically estimate the amount of information in different samples at different scales. , ; to obtain information at different scales Then, by comparing with the encoded feature vector The dot product yields the feature representation at that scale, and then the feature representations at different scales are concatenated to obtain a multi-scale fused feature vector; S3. Use the dataset to train the multi-source time series signal classification model to obtain the trained multi-source time series signal classification model; S4. Use the trained multi-source time series signal classification model to classify the input multi-source sensor time series signals.

2. The multi-source time series signal classification method based on multi-scale feature fusion according to claim 1, characterized in that, In the signal preprocessing module, the acquired multi-source sensor time series signals are preprocessed, including filtering, denoising, and smoothing, to remove noise signals generated during data acquisition.

3. The multi-source time series signal classification method based on multi-scale feature fusion according to claim 1, characterized in that, In the multi-scale feature extractor, the preprocessed multi-source sensor time series signal is divided into time subsequences of different lengths on different scale branches. The time domain, frequency domain, time-frequency domain and location feature information of the corresponding time subsequences of different scale branches are extracted respectively, and then spliced ​​to generate the feature embedding of the corresponding time subsequence. The features of multiple time subsequences of different lengths are embedded and input into an encoder of a set scale for feature encoding to obtain a multi-scale feature representation of the preprocessed multi-source sensor time series signal.

4. The multi-source time series signal classification method based on multi-scale feature fusion according to claim 1, characterized in that, After obtaining the multi-scale fusion feature vectors of time series signals from multiple sources, in order to fully explore the correlation between time series signals from different sources, the cross-attention fusion module uses the cross-attention fusion mechanism to fuse the multi-scale fusion feature vectors of time series signals from different sources to obtain robust signal representations.

5. The multi-source time series signal classification method based on multi-scale feature fusion according to claim 1, characterized in that, In step S3, based on the total loss function A multi-source time series signal classification model was trained using a dataset; By maximizing the Kullback-Leibler divergence between the predicted and true distributions For the second classifier Conduct training: in, Indicates the first The second classifier of each branch The output of the first The probability distribution of each sample; In order to train the information-aware network Using L2 loss : ； The total loss function + + ; This represents the overall classification loss of a multi-source time series signal classification model.

6. The multi-source time series signal classification method based on multi-scale feature fusion according to claim 5, characterized in that, Overall classification loss of multi-source time series signal classification model The cross-entropy loss function is used.

7. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the multi-source time series signal classification method based on multi-scale feature fusion as described in any one of claims 1 to 6.

Images