A time series data new class discovery method based on multi-modal fusion

Abstract

The application discloses a kind of based on multimodal fusion's time series data new class discovery method, comprising: S1: obtaining initial time series data, carries out enhancement processing, obtains enhanced time series data;S2: respectively to initial time series data and enhanced time series data are carried out conversion processing, obtain line chart;S3: establish time series data new class discovery network model, to time series data new class discovery network model training, obtain trained time series data new class discovery network model;Time series data new class discovery network model includes multimodal feature extraction module, multimodal feature fusion module and parameter type classifier;S4: based on trained time series data new class discovery network model and actual acquisition time series data are classified to obtain classification result.The application can effectively improve the unknown class time series data classification precision, while retaining the effective classification ability to known class time series data, so that time series data classification is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer time series data classification, and particularly relates to a time series data new class discovery method based on multi-modal fusion. BACKGROUND

[0002] The classification of time series data new class discovery is a method for classifying known class labels of target objects and unknown class labels according to different features reflected by labeled data and unlabeled data in time series data under the condition that part of the data lacks data labeling. Time series data classification is one of the core tasks in computer systems, and is widely used in human activity recognition, medical electrocardiogram monitoring, and abnormal detection and many other fields. New class discovery is a new type of semi-supervised learning paradigm, that is, through the use of labeled samples, the classification task of unlabeled data is completed. The main difficulty in the classification process of new class discovery is the prediction of time series data that has not been seen. Due to the characteristics of multivariate of time series data and the small difference between classes, the classification of time series data is a great challenge.

[0003] The earliest research on new class discovery is NCD (K. Han, A. Vedaldi, and A. Zisserman. Learning to discover novel visual categories via deep transfer clustering. In IEEE International Conference on Computer Vision, ICCV, pages. 8400-8408, 2019), which proposes that the model can discover new classes by simulating human vision, just like a child can easily discover new animal classes after learning some animal classes. In addition, on the basis of new class discovery, generalized new class discovery is proposed, that is, in the unlabeled data, not only unknown classes of data, but also known classes of data are contained, GCD (S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, “Generalized category discovery,” In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 7492-7501), but these methods are focused on the research in the field of computer vision, and there are few related works on new class discovery of time series data. At present, the only method MRNCL (M. Zhang, T. Zhu, M. Nie, and Z. Liu, “More reliable neighborhood contrastive learning for novel class discovery in sensor-based human activity recognition,” Sensors, vol. 23, no. 23, p. 9529, 2023) explores new human activity classes through reliable neighborhood contrastive learning, but this method does not consider the case of known class data in unlabeled data. Therefore, how to complete the classification of known classes and unknown classes in unlabeled data is a problem to be solved. SUMMARY

[0004] The application provides a time series data new class discovery method based on multi-modal fusion, to overcome the technical problems of the existing time series data new class discovery method in distinguishing known classes and unknown classes.

[0005] In order to achieve the above object, the technical scheme of the present application is:

[0006] A time series data new class discovery method based on multi-modal fusion, the specific steps comprising:

[0007] S1: obtaining initial time series data, performing enhancement processing on the initial time series data to obtain enhanced time series data; the initial time series data includes a labeled data set composed of visible classes and an unlabeled data set containing all classes;

[0008] S2: converting the initial time series data and the enhanced time series data respectively to obtain a first line chart and a second line chart;

[0009] S3: establishing a time series data new class discovery network model, training the time series data new class discovery network model based on the initial time series data, the enhanced time series data, the first line chart and the second line chart, and obtaining the trained time series data new class discovery network model when the set total loss function converges;

[0010] The time series data new class discovery network model comprises a multi-modal feature extraction module, a multi-modal feature fusion module and a parameter type classifier;

[0011] The multi-modal feature extraction module is used to extract time series features from the initial time series data and the enhanced time series data respectively through a double-branch structure, and extract image features from the first line chart and the second line chart;

[0012] The multi-modal feature fusion module is used to perform feature fusion on the time series features and the image features respectively based on a multi-head attention fusion mechanism to obtain first fusion features and second fusion features;

[0013] The parameter type classifier is used to obtain a classification result based on the first fusion features and the second fusion features;

[0014] S4: performing classification based on the trained time series data new class discovery network model and the actually obtained time series data to obtain a classification result.

[0015] Further, the multi-modal feature extraction module comprises a numerical time series data feature extraction module and an image type time series data feature extraction module;

[0016] The specific steps of extracting time series features from the initial time series data and the enhanced time series data and extracting image features from the first line chart and the second line chart through the double-branch structure of the multi-modal feature extraction module comprise:

[0017] The first time sequence feature and the second time sequence feature are extracted from the initial time sequence data and the enhanced time sequence data respectively by the numerical time sequence data feature extraction module;

[0018] The first image feature and the second image feature are extracted from the first line graph and the second line graph by the image time sequence data feature extraction module.

[0019] Further, the multi-modal feature fusion module comprises a first time sequence feature attention fusion module, a first image feature attention fusion module and a first gate multi-modal fusion module;

[0020] The multi-modal feature fusion module performs feature fusion on the first time sequence feature and the first image feature based on a multi-head attention fusion mechanism, and the specific steps of obtaining the first fusion feature comprise:

[0021] The first time sequence feature and the first image feature are subjected to first multi-head attention feature fusion by the first time sequence feature attention fusion module, and the fusion result is transmitted to the first gate multi-modal fusion module;

[0022] The first time sequence feature and the first image feature are subjected to second multi-head attention feature fusion by the first image feature attention fusion module, and the fusion result is transmitted to the first gate multi-modal fusion module;

[0023] The first gate multi-modal fusion module generates fusion weights based on the outputs of the first time sequence feature attention fusion module and the first image feature attention fusion module, and performs feature fusion on the outputs of the first time sequence feature attention fusion module and the first image feature attention fusion module based on the fusion weights to obtain the first fusion feature.

[0024] Further, the multi-modal feature fusion module further comprises a second time sequence feature attention fusion module, a second image feature attention fusion module and a second gate multi-modal fusion module;

[0025] The multi-modal feature fusion module performs feature fusion on the second time sequence feature and the second image feature based on a multi-head attention fusion mechanism, and the specific steps of obtaining the second fusion feature comprise:

[0026] The second time sequence feature and the second image feature are subjected to second multi-head attention feature fusion by the second time sequence feature attention fusion module, and the fusion result is transmitted to the second gate multi-modal fusion module;

[0027] The second time sequence feature and the second image feature are subjected to second multi-head attention feature fusion by the second image feature attention fusion module, and the fusion result is transmitted to the second gate multi-modal fusion module;

[0028] The second gated multimodal fusion module generates fusion weights based on the outputs of the second temporal feature attention fusion module and the second image feature attention fusion module. Based on the fusion weights, the outputs of the second temporal feature attention fusion module and the second image feature attention fusion module are fused to obtain the second fused feature.

[0029] Furthermore, the total loss function Including contrastive learning loss function and classification loss function , represented as:

[0030] (1)

[0031] (2)

[0032] (3)

[0033] in, This indicates that there is a supervised comparison of losses. The balance between unsupervised contrast loss, supervised contrast loss, and unsupervised contrast loss is represented by parameters. To control; It is a hyperparameter that controls the balance between the two loss functions; This indicates supervised classification loss. The balance between unsupervised classification loss, supervised classification loss, and unsupervised classification loss is represented by parameters. To control.

[0034] Furthermore, the formula for unsupervised contrastive loss is expressed as:

[0035] (4)

[0036] in, This indicates the number of data points in a training batch. That is, the first i Feature representation of a sample For the corresponding augmented feature representation, and Both are considered as positive sample pairs; That is, other feature representations in the same training batch of data. It is the temperature coefficient;

[0037] The formula for supervised contrast loss is expressed as:

[0038] (5)

[0039] in, is the number of labeled samples in a training batch, is the temperature coefficient; is the feature representation of the th sample; is the set of samples with the same label in the same training batch.

[0040] Further, the formula of the unsupervised classification loss is:

[0041] (6)

[0042] where, is the self-distillation classification loss, is the regularization loss; is the cross-entropy loss function; is the predicted probability value corresponding to the input time series data, is the probability value output by the augmented time series data sample;

[0043] The formula of the regularization loss is as follows:

[0044] (7)

[0045] (8)

[0046] where, is the probability value output by the augmented time series data sample, is the average predicted probability value output in a training batch;

[0047] The formula of the supervised classification loss function is:

[0048] (9)

[0049] where, is the cross-entropy loss function, is the number of labeled samples in a training batch, is the true label corresponding to the input sample i .

[0050] Beneficial effects: the application establishes a time series data new class discovery network model, and trains it through initial time series data including a labeled data set and an unlabeled data set, extracts time series features from the initial time series data and enhanced time series data through the double-branch structure of the constructed multi-modal feature extraction module, and extracts image features from the first and second line graphs; the multi-modal feature fusion module is used to fuse the time series features and the image features, thereby improving the discrimination ability between known categories and unknown categories, thereby improving the time series data classification accuracy, especially the classification accuracy of the unseen categories. In summary, the application can effectively improve the classification accuracy of the unknown class time series data, while retaining the effective classification ability of the known class time series data, making the time series data classification more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0052] Figure 1 The flow chart of a time series data new class discovery method based on multi-modal fusion in the application;

[0053] Figure 2 The model diagram of a time series data new class discovery method based on multi-modal fusion in the embodiment of the application;

[0054] Figure 3 The classification task diagram of time series data new class discovery set in the embodiment of the application;

[0055] Figure 4 The loss function hyperparameter in the training process of the time series data new class discovery network model in the embodiment of the application The line graph of the experimental results;

[0056] Figure 5 The comparison diagram of the experimental results in the embodiment of the application. DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0058] The embodiment provides a time series data new class discovery method based on multi-modal fusion, as shown in Figure 1 and Figure 2 , and the specific steps include:

[0059] S1: obtaining initial time series data, performing enhancement processing on the initial time series data to obtain enhanced time series data; as shown in Figure 3 , the initial time series data includes a labeled data set composed of visible classes and an unlabeled data set containing all classes;

[0060] S2: performing conversion processing on the initial time series data and the enhanced time series data respectively to obtain a first line chart and a second line chart;

[0061] Specifically, the obtained initial time series data is subjected to data enhancement to obtain enhanced time series data , the initial time series data is converted into a corresponding line chart , the enhanced time series data is converted into a corresponding line chart ; for each variable in the initial time series data , a line chart is drawn, and all line charts are spliced into an image to obtain a corresponding first line chart of the initial time series data , and a corresponding second line chart of the enhanced time series data is obtained in the same way.

[0062] Specifically, the embodiment adds Gaussian noise to the initial time series data to obtain the enhanced time series data .

[0063] In specific embodiments, numerical time series data is converted into a line chart image, and the method is as follows:

[0064] For time series data Each variable draws a line chart, for example: the time series data of the DSA data set sample has 45 variables, a line chart is drawn for each variable, and all line charts are spliced into an image, that is, a 7*7 square chart with a size of 512*512 Similarly, the enhanced time series data The corresponding line chart The embodiment adopts the method of converting time series data into images, which can better capture the trend, time pattern and other information of time series data, and is helpful to the subsequent new class discovery process.

[0065] S3: Establish a time series data new class discovery network model (TimeGCD), train the time series data new class discovery network model based on the initial time series data, enhanced time series data, first line chart and second line chart, and obtain the trained time series data new class discovery network model when the set total loss function converges;

[0066] The time series data new class discovery network model comprises a multi-modal feature extraction module, a multi-modal feature fusion module and a parameter type classifier.

[0067] The multi-modal feature extraction module is used to extract time series features from the initial time series data and enhanced time series data respectively through a double-branch structure, and extract image features from the first line chart and second line chart;

[0068] The multi-modal feature fusion module is used to perform feature fusion on the time series features and image features respectively based on a multi-head attention fusion mechanism to obtain first fusion features and second fusion features;

[0069] The parameter type classifier is used to obtain a classification result based on the first fusion features and second fusion features;

[0070] S4: Based on the trained time series data new class discovery network model and the actually obtained time series data, classification is performed to obtain a classification result.

[0071] Specifically, the trained time series data new class discovery network model can realize sample classification of visible classes and invisible classes.

[0072] In specific embodiments, the multi-modal feature extraction module comprises a numerical time series data feature extraction module and an image time series data feature extraction module.

[0073] The specific steps that the multi-modal feature extraction module extracts time series features from the initial time series data and enhanced time series data respectively through a double-branch structure, and extracts image features from the first line chart and second line chart include:

[0074] The numerical time series data feature extraction module extracts the first time series feature and the second time series feature, i.e., the global time trend, from the initial time series data and the enhanced time series data, respectively.

[0075] The image-type time-series data feature extraction module extracts the first image features and the second image features, namely local waveform shape features, from the first line chart and the second line chart.

[0076] Specifically, the numerical time-series data feature extraction module is a time-series feature extractor. The temporal feature extractor comprises multiple stacked one-dimensional convolutional layers with expanded receptive fields, and uses a primitive numerical temporal data feature extraction method to extract numerical first temporal features from the initial temporal data and the enhanced temporal data. Second time series characteristics The operating formula for a one-dimensional convolutional layer is as follows:

[0077] (1)

[0078] in, This represents the first time series operation applied to a time series. t The result of each element Represents position i The convolution kernel used is b Inflation factor is d The value after convolution.

[0079] Specifically, the image-based time-series data feature extraction module is an image feature extractor. In this embodiment, a ResNet50 network is preferably used as the image feature extractor. The image feature extractor adopts a primitive image-type time-series data feature extraction method, starting from the first piecewise linear graph. Second line chart The image-type temporal features, i.e., the first image features, are extracted from them. Second image features .

[0080] In a specific embodiment, the multimodal feature fusion module includes a first temporal feature attention fusion module, a first image feature attention fusion module, and a first gated multimodal fusion module;

[0081] The multimodal feature fusion module performs feature fusion on the first temporal feature and the first image feature based on a multi-head attention fusion mechanism to obtain the first fused feature. The specific steps include:

[0082] The first timing feature attention fusion module is configured to perform first multi-head attention feature fusion on the first timing feature and the first image feature, and transmit a fusion result to the first gate multi-modal fusion module.

[0083] The first image feature attention fusion module is configured to perform second multi-head attention feature fusion on the first timing feature and the first image feature, and transmit a fusion result to the first gate multi-modal fusion module.

[0084] The first gate multi-modal fusion module is configured to generate a fusion weight based on outputs of the first timing feature attention fusion module and the first image feature attention fusion module, and perform feature fusion on the outputs of the first timing feature attention fusion module and the first image feature attention fusion module based on the fusion weight to obtain a first fusion feature.

[0085] In specific embodiments, the multi-modal feature fusion module further comprises a second timing feature attention fusion module, a second image feature attention fusion module, and a second gate multi-modal fusion module.

[0086] The multi-modal feature fusion module performs feature fusion on the second timing feature and the second image feature based on a multi-head attention fusion mechanism, and the specific steps of obtaining a second fusion feature include:

[0087] The second timing feature attention fusion module is configured to perform second multi-head attention feature fusion on the second timing feature and the second image feature, and transmit a fusion result to the second gate multi-modal fusion module.

[0088] The second image feature attention fusion module is configured to perform second multi-head attention feature fusion on the second timing feature and the second image feature, and transmit a fusion result to the second gate multi-modal fusion module.

[0089] The second gate multi-modal fusion module is configured to generate a fusion weight based on outputs of the second timing feature attention fusion module and the second image feature attention fusion module, and perform feature fusion on the outputs of the second timing feature attention fusion module and the second image feature attention fusion module based on the fusion weight to obtain a second fusion feature.

[0090] Specifically, for multi-modal feature fusion, it mainly consists of two parts, one is a double-branch multi-head attention fusion, and the other is a gate multi-modal fusion module to control the weight ratio of the fusion of two modalities.

[0091] Specifically, the multi-modal fusion process based on the first timing feature attention fusion module, the first image feature attention fusion module, and the first gate multi-modal fusion module includes:

[0092] input the first time sequence feature and the first image feature into the first time sequence feature attention fusion module and the first image feature attention fusion module;

[0093] S201: performing time sequence feature attention fusion through the first time sequence feature attention fusion module, including: inputting the first time sequence feature as the query q of the multi-head attention, the first image feature as the key-value pair k / v, fusing the fused feature with the first time sequence feature after residual connection to obtain the attention fusion feature , and transmitting the attention fusion feature to the first gate multi-modal fusion module;

[0094] S202: performing image feature attention fusion through the first image feature attention fusion module, including: inputting the first image feature as the query q of the multi-head attention, the first time sequence feature as the key-value pair k / v, fusing the fused feature with the first image feature after residual connection to obtain the attention fusion feature , and transmitting the attention fusion feature to the first gate multi-modal fusion module;

[0095] Specifically, in S201-S202, the embodiment respectively obtains the fusion feature through two multi-head attention branches, and the formula of the multi-head attention mechanism is as follows:

[0096] (2)

[0097] (3)

[0098] (4)

[0099] Specifically, in the multi-head attention mechanism, the input is composed of the query matrix Q, the key matrix K and the value matrix V. Through the learnable projection matrix and , each attention head maps the input to the subspace of the key / query (Q / K) dimension and the value dimension (V) respectively. After the outputs of all attention heads are combined through the splicing operation , the dimension is restored by the output projection matrix , h is the number of attention heads.

[0100] The formula of the attention fusion feature and is as follows:

[0101] (5)

[0102] (6)

[0103] wherein, is multi-head attention, is normalization layer, is time sequence feature, is image feature.

[0104] S203: fusing the two attention fusion features according to the obtained attention fusion features and the fusion weight G obtained in S202 to obtain a first fusion feature . ;

[0105] Specifically, the gating multi-modal fusion module is used to control the fusion proportion of the attention fusion features of the two modalities, which balances the features of the two modalities by receiving the feature values of the two modalities and generating gating weights through a feedforward neural network.

[0106] Specifically, after the attention fusion features and are merged, a fusion weight vector G is learned through a feedforward neural network, and the two attention fusion features are fused according to the fusion weight G to obtain a fusion feature , and finally the fusion feature is input to the feedforward neural network and connected in residual connection with the input, and then a normalization operation is performed to obtain a first fusion feature , which is expressed as follows:

[0107] (7)

[0108] (8)

[0109] (9)

[0110] wherein, FFN represents a feedforward neural network including multiple linear layers Linear and ReLU layers and a Sigmoid layer, represents the multiplication of vectors.

[0111] Specifically, for the enhanced features and , the embodiment adopts a similar process as S201~S203 to obtain a second fusion feature , which is not described here.

[0112] Specifically, the obtained first fusion feature is input to the parametric classifier to obtain a predicted label​ .

[0113] In specific embodiments, for the whole network end-to-end training, the total loss function includes a contrastive learning loss function and a classification loss function , both of which are composed of supervised loss and unsupervised loss, denoted as:

[0114] (10)

[0115] (11)

[0116] (12)

[0117] wherein, represents the supervised contrastive loss, represents the unsupervised contrastive loss, and the balance between the supervised contrastive loss and the unsupervised contrastive loss is controlled by the parameter ; is a hyperparameter that controls the balance between the two loss functions; represents the supervised classification loss, represents the unsupervised classification loss, and the balance between the supervised classification loss and the unsupervised classification loss is controlled by the parameter .

[0118] Specifically, as shown in Figure 4 , when takes the value of 1, the overall accuracy of TimeGCD reaches the highest, proving that taking the value of 1 can balance the two loss functions.

[0119] In specific embodiments, the formula of the unsupervised contrastive loss is:

[0120] (13)

[0121] wherein, represents the number of data in a training batch, is the feature representation of the i th sample, is the feature representation after data augmentation corresponding to the i th sample, and both serve as positive sample pairs; that is, the feature representations of other samples in the same batch of data, as negative sample pairs, is the temperature coefficient;

[0122] Specifically, the unsupervised contrastive loss The role is to pull the distance of positive sample pairs closer and push the distance of negative sample pairs farther in the feature space.

[0123] Specifically, the supervised contrastive loss function is similar, the difference is that the selection of positive and negative sample pairs is determined according to the label. For data in a batch, feature representations of samples with the same label are positive sample pairs, and feature representations of samples with different labels are negative sample pairs. The formula of the supervised contrastive loss is:

[0124] (14)

[0125] wherein, denotes the number of labeled samples in a training batch, is a temperature coefficient; is the feature representation of the i-th sample; is the feature representation of the i-th sample; is the set of samples with the same label in the same training batch.

[0126] In specific embodiments, the formula of the unsupervised classification loss is:

[0127] (15)

[0128] wherein, is the self-distillation classification loss, is the regularization loss; is a cross-entropy loss function; is the predicted probability value corresponding to the input time series data, is the probability value output by the sample after time series data augmentation;

[0129] The regularization loss is used to constrain unsupervised samples so that the label region of the model classification output is averaged. The formula of the regularization loss is as follows:

[0130] (16)

[0131] (17)

[0132] wherein, is the probability value output by the sample after time series data augmentation, denotes the average predicted probability value output in a batch;

[0133] Specifically, the formula of the supervised classification loss function is:

[0134] (18)

[0135] wherein, is a cross-entropy loss function, denotes the number of labeled samples in a training batch, is an input sample i corresponding real label.

[0136] Specifically, in the present embodiment, for the unlabeled data, the first fusion feature and the corresponding second fusion feature are calculated as the only positive sample pair, and the unsupervised contrastive learning loss function is calculated. For the labeled data, the positive and negative sample pairs are constructed by the label value of the labeled data, and the supervised contrastive learning loss function is calculated.

[0137] Specifically, the trained time series data new class discovery network model is used to test the image, the time series data is input into the TimeGCD, and finally a prediction score is obtained through the parametric classifier, and the label with the maximum score is selected as the time series prediction label.

[0138] Specifically, for known classes, the time series data new class discovery network model can fully identify different classes through the training of the real value label; for unknown classes, the time series data new class discovery network model obtains the feature vector through the encoder trained with the knowledge of known classes, and further through the contrastive learning between the sample and the enhanced view, similar unknown classes are gathered together in the feature space, and the feature vectors of different classes are far away, and finally the parametric classifier is used to realize the classification of known and unknown classes.

[0139] In order to verify the effectiveness of the double-branch design and multi-modal fusion method in TimeGCD, the present embodiment respectively uses single-modal data for experimental comparison, and adopts an element-level addition fusion module for comparison. The experimental results are shown in Figure 5 The accuracy of the method proposed in the present embodiment on the double-modal branch is higher than that of the single-branch mode, and the effect of the designed gating multi-modal fusion module is better than that of the element-level addition fusion, especially the accuracy of the new class, and the accuracy improvement effect is more obvious.

[0140] The method proposed in the present embodiment is compared with the most advanced new class discovery classification method (GCD, SimGCD, PCR, and CMS four methods). Compared with the four most advanced methods, the total accuracy index of the method proposed in the present embodiment on the two most common data sets DSA and PAMAP2 reaches the most advanced performance.

[0141] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for discovering new categories in time series data based on multimodal fusion, characterized in that, The specific steps include: S1: Obtain initial time-series data, perform enhancement processing on the initial time-series data to obtain enhanced time-series data; the initial time-series data includes a labeled data set consisting of visible categories and an unlabeled data set containing all categories; S2: The initial time series data and the enhanced time series data are transformed to obtain the first line chart and the second line chart; S3: Establish a new class discovery network model for time series data. Train the new class discovery network model for time series data based on the initial time series data, augmented time series data, the first line chart and the second line chart. When the set total loss function converges, the trained new class discovery network model for time series data is obtained. The time-series data new class discovery network model includes a multimodal feature extraction module, a multimodal feature fusion module, and a parametric classifier; The multimodal feature extraction module is used to extract temporal features from the initial temporal data and the enhanced temporal data respectively through a dual-branch structure, and to extract image features from the first line graph and the second line graph; The multimodal feature fusion module is used to perform feature fusion on the temporal features and image features respectively based on the multi-head attention fusion mechanism to obtain the first fused feature and the second fused feature; The parametric classifier is used to obtain a classification result based on the first fusion feature and the second fusion feature; S4: Classify the time series data based on the trained new class discovery network model and the actual acquired time series data to obtain the classification results.

2. The method for discovering new categories of time series data based on multimodal fusion according to claim 1, characterized in that, The multimodal feature extraction module includes a numerical time-series data feature extraction module and an image-based time-series data feature extraction module; The multimodal feature extraction module extracts temporal features from the initial temporal data and the enhanced temporal data respectively through a dual-branch structure. The specific steps for extracting image features from the first and second line graphs include: The numerical time series data feature extraction module extracts the first time series feature and the second time series feature from the initial time series data and the enhanced time series data, respectively. The image-based time-series data feature extraction module extracts the first image features and the second image features from the first and second line charts.

3. The method for discovering new categories of time series data based on multimodal fusion according to claim 2, characterized in that, The multimodal feature fusion module includes a first temporal feature attention fusion module, a first image feature attention fusion module, and a first gated multimodal fusion module; The multimodal feature fusion module performs feature fusion on the first temporal feature and the first image feature based on a multi-head attention fusion mechanism to obtain the first fused feature. The specific steps include: The first temporal feature attention fusion module performs a first multi-head attention feature fusion on the first temporal feature and the first image feature, and transmits the fusion result to the first gated multimodal fusion module. The first image feature attention fusion module performs a second multi-head attention feature fusion on the first temporal feature and the first image feature, and transmits the fusion result to the first gated multimodal fusion module. The first gated multimodal fusion module generates fusion weights based on the outputs of the first temporal feature attention fusion module and the first image feature attention fusion module. Based on the fusion weights, the outputs of the first temporal feature attention fusion module and the first image feature attention fusion module are fused to obtain the first fused feature.

4. The method for discovering new categories of time series data based on multimodal fusion according to claim 3, characterized in that, The multimodal feature fusion module further includes a second temporal feature attention fusion module, a second image feature attention fusion module, and a second gated multimodal fusion module; The multimodal feature fusion module fuses the second temporal feature and the second image feature based on a multi-head attention fusion mechanism to obtain the second fused feature. The specific steps include: The second temporal feature attention fusion module performs a second multi-head attention feature fusion on the second temporal feature and the second image feature, and transmits the fusion result to the second gated multimodal fusion module. The second image feature attention fusion module performs a second multi-head attention feature fusion on the second temporal feature and the second image feature, and transmits the fusion result to the second gated multimodal fusion module. The second gated multimodal fusion module generates fusion weights based on the outputs of the second temporal feature attention fusion module and the second image feature attention fusion module. Based on the fusion weights, the outputs of the second temporal feature attention fusion module and the second image feature attention fusion module are fused to obtain the second fused feature.

5. The method for discovering new categories of time series data based on multimodal fusion according to claim 1, characterized in that, The total loss function Including contrastive learning loss function and classification loss function , represented as: (1) (2) (3) in, This indicates that there is a supervised comparison of losses. The balance between unsupervised contrastive loss, supervised contrastive loss, and unsupervised contrastive loss is represented by parameters. To control; It is a hyperparameter that controls the balance between the two loss functions; This indicates supervised classification loss. The balance between unsupervised classification loss, supervised classification loss, and unsupervised classification loss is represented by parameters. To control.

6. The method for discovering new categories of time series data based on multimodal fusion according to claim 5, characterized in that, The formula for unsupervised contrast loss is expressed as: (4) in, This indicates the number of data points in a training batch. That is, the first i Feature representation of a sample For the corresponding augmented feature representation, and Both are considered as positive sample pairs; That is, other feature representations in the same training batch of data. It is the temperature coefficient; The formula for supervised contrast loss is expressed as: (5) in, This indicates the number of labeled samples in a training batch. Temperature coefficient; For the first Feature representation of each sample; This refers to the set of samples with the same label within the same training batch.

7. The method for discovering new categories of time series data based on multimodal fusion according to claim 6, characterized in that, The formula for unsupervised classification loss is expressed as: (6) in, It is a loss due to self-distillation and separation. It is the regularization loss; The cross-entropy loss function; This represents the predicted probability value corresponding to the input time series data. This represents the probability value of the sample output after time-series data augmentation. Regularization loss The formula is expressed as follows: (7) (8) in, This represents the probability value of the sample output after time-series data augmentation. This represents the average predicted probability value of the output within a training batch; Supervised classification loss function The formula is: (9) in, Let cross-entropy be the loss function. This indicates the number of labeled samples in a training batch. For input samples i The corresponding real tags.

Images