Semi-supervised time sequence regression system based on potential spatial dimension mask reconstruction

The semi-supervised timing regression system through latent spatial dimension mask reconstruction uses pre-trained comparative learning and Crossformer encoder, combined with timing and signal strength data enhancement, and solves the performance of the existing semi-supervised timing regression method in the case of sparse label data by using pre-trained comparative learning and Crossformer encoder, combining timing and signal strength data enhancement, which solves the performance of the existing semi-supervised timing regression method in the case of sparse label data. Improve the problem and achieve higher prediction accuracy and timing regression performance.

CN119046896APending Publication Date: 2024-11-29ZHEJIANG SCI-TECH UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411031573.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing semi-supervised timing regression method is difficult to improve performance when there are few label data, especially based on the generative semi-supervised and self-supervised comparison learning method, there are training difficulties and convergence problems, and it is impossible to effectively utilize unlabeled data.

Method used

Using a semi-supervised timing regression system based on latent spatial dimension mask reconstruction, the timing and signal strength data enhancement are carried out to capture complex dependencies between dimensions through pre-training comparison learning, Crossformer encoder and dual-view mask prediction comparison loss tasks. , and remove the comparison learning loss in the optimization task, and only the double-view mask prediction task is retained.

Benefits of technology

The prediction regression accuracy in a small number of labeled data scenarios is improved, and the performance of timing regression tasks is improved through meticulous data feature learning and dimensional level representation, for example, the prediction accuracy of CV value is improved in industrial chemical fiber data scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046896A_ABST
    Figure CN119046896A_ABST
Patent Text Reader

Abstract

The invention provides a semi-supervised time sequence regression system based on potential spatial dimension mask reconstruction, and aims to solve the problem of time sequence analysis under the condition of scarcity of marked data. The system comprises an original data enhancement module, a cross-dimension space coding module and an optimization task module, the original data enhancement module is used for performing data enhancement on original data in time sequence and signal intensity, effectively learning time sequence dependence and cross-dimension dependence of the data, and generating stable enhanced view codes; the cross-dimension space coding module is used for carrying out fine representation of dimension levels and capturing a complex dependency relationship between dimensions; the task optimization module adopts a double-view mask prediction task which is more beneficial to a downstream regression task, can be effectively applied to a semi-supervised time sequence regression problem, and improves the model performance in a scene with rare marked data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of time series data and deep learning technology, and specifically relates to a semi-supervised time series regression technology solution (HSMC) based on latent space dimensional mask reconstruction. Background Art

[0002] Time series data processing refers to the process of analyzing and processing data arranged in chronological order. These data may come from various sensors, devices, systems or applications, and are used to monitor, analyze and predict the changing trends of specific events or phenomena. Time series data processing has a wide range of applications in many fields, such as finance, medical care, energy, transportation, etc. There are two mainstream technologies in the time series field - time series classification and time series regression.

[0003] Time series regression, also known as time series forecasting, is a technology widely used in finance, e-commerce, tourism and other industries. It uses historical data to predict future development trends by analyzing the time correlation of data. Time series forecasting methods are mainly divided into two categories: traditional machine learning methods and deep learning methods. Traditional methods include ARIMA and Prophet algorithms. The ARIMA model logarithmically transforms or differentially fits the data through the mean and variance invariance of time series data. The Prophet algorithm decomposes the time series into seasonal, trend and residual series, and combines machine learning technology for fitting, which is particularly suitable for time series with strong seasonal effects. Deep learning methods are based on recurrent neural networks (RNNs) and attention mechanisms. RNNs are widely used in time series forecasting due to their strong ability to capture time series dependencies. The Deep AR model predicts scalar time series by combining RNNs and autoregressive models. In order to address the limitations of RNNs in capturing long-term dependencies, researchers have introduced Transformer architectures such as Informer and AST, which improve the performance of models in long-term forecasting and sparse time series forecasting through different innovative designs.

[0004] Semi-supervised learning (SSL) is a machine learning method that combines the characteristics of supervised learning and unsupervised learning. In SSL, the dataset consists of labeled data and unlabeled data, of which the labeled data usually accounts for only 1% to 10% of the total. The goal of SSL is to use unlabeled data to improve model performance, which is based on three key assumptions: semi-supervised smoothness assumption, clustering assumption, and manifold assumption.

[0005] Semi-supervised regression (SSR) is an application of semi-supervised learning, which aims to train a regression model using labeled and unlabeled data to predict continuous values. SSR methods are mainly divided into two types: non-parametric and parametric. Non-parametric methods, such as regression based on a common training paradigm, kernel regression, and Laplace regularized regression, have good flexibility. Parametric methods extend supervised regression methods to semi-supervised regression, such as linear regression and ridge regression. Although these methods have achieved certain success in semi-supervised learning, they are still limited by the complexity of machine learning itself, resulting in the algorithm performance not yet reaching a satisfactory level.

[0006] Semi-supervised time series learning is an application of SSL in the field of time series data, which aims to improve model performance by utilizing unlabeled time series data. Semi-supervised time series classification can be divided into methods based on traditional machine learning and deep learning. Traditional methods such as DTW-D and Divide Mix use dynamic time warping distance and dual-view combination methods to infer the labels of unlabeled data. Deep learning methods such as Semi Time and TSTCC improve model performance by reconstructing data distribution or contrastive learning.

[0007] Semi-supervised time series regression is an extension of semi-supervised time series classification. There is little research on it currently, and it is usually trained based on pre-trained models. Some methods, such as BSTF, TNC, and Tsev2e, learn unlabeled data representations through self-supervised contrastive loss and then fine-tune them in downstream tasks. There are also studies that use adversarial generative networks to solve regression problems, such as the REG-GAN framework and the feature contrast loss function proposed by SRGAN.

[0008] In general, semi-supervised learning improves the generalization ability and performance of the model by effectively utilizing unlabeled data. In many fields such as classification, regression and time series analysis, SSL has shown its unique value and potential. Although there are some challenges, such as the complexity of model training and the difficulty of convergence, with the deepening of research, SSL is expected to be applied and promoted in more fields.

[0009] At present, there are two main types of methods based on semi-supervised time series regression. One type is based on traditional machine learning, which flattens time series data into two dimensions and then uses traditional semi-supervised machine learning methods for learning. The other type is based on deep learning methods, which can be further divided into generative semi-supervised time series regression and self-supervised contrastive learning.

[0010] Generative semi-supervised time series regression uses unlabeled data to learn the essential characteristics of data through discriminators and generators, but this method is limited by the problem that adversarial generative network training is difficult to converge, so it cannot be applied to all scenarios. The self-supervised contrastive learning method is based on unlabeled contrastive learning. In the first stage, data enhancement is performed on unlabeled data to construct positive and negative samples, thereby learning a common data feature representation, and then fine-tuning it in downstream tasks. Although this method can effectively utilize unlabeled data, the model performance is still unsatisfactory when labeled data is extremely scarce. Summary of the invention

[0011] Based on the above situation, the present invention proposes a semi-supervised temporal regression system based on mask reconstruction of latent space dimensions.

[0012] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0013] A semi-supervised temporal regression system based on latent space dimensional mask reconstruction, using a pre-trained contrastive learning training method, mainly including an original data data enhancement module, a cross-dimensional space encoding module and an optimization task module;

[0014] The raw data enhancement module is used to perform data enhancement on the raw data in terms of timing and signal strength, respectively, to generate stable enhanced view coding;

[0015] The cross-dimensional spatial encoding module uses the Crossformer encoder to achieve refined representation at the dimensional level and capture the complex dependencies between dimensions. The present invention introduces the Crossformer encoder for feature extraction, which overcomes the limitations of the traditional Transformer architecture, can learn to distinguish the potential interdependencies between dimensions, and refine the representation at the dimensional level;

[0016] The optimization task module of the present invention removes the contrastive learning loss widely used in time series classification tasks, and only retains the dual-view mask prediction contrastive loss task, so as to better improve the performance of the time series regression task.

[0017] Preferably, the raw data enhancement module is used to perform data enhancement on the raw data in terms of timing and signal strength respectively. In this module, the present invention regards the Shift operation as a weak enhancement in terms of timing and the Drift as a strong enhancement in terms of signal strength through corresponding data enhancement parameters, thereby achieving the requirements of strong and weak view data enhancement.

[0018] The cross-dimensional space encoding module uses the Crossformer encoder to achieve refined representation at the dimension level and capture the complex dependencies between dimensions. The cross-dimensional space encoding module introduces the Crossformer encoder for feature extraction, which overcomes the limitations of the traditional Transformer architecture, can effectively distinguish the potential interdependencies between dimensions, and refine the representation at the dimension level.

[0019] The optimization task module described above removes more contrastive learning losses used in time series classification tasks, and only retains the dual-view mask prediction task to better improve the performance of time series regression tasks. The detailed model framework can be seen below Figure 1 shown.

[0020] Preferably, the original data enhancement module includes two stages: timing enhancement and signal enhancement.

[0021] Preferably, in the timing enhancement stage, the present invention adopts a Shift operation with weak timing enhancement to construct positive samples by shifting the original data in the timing arrangement, and it is expected that the model can effectively capture the essential characteristics of the original data in terms of timing.

[0022] Specifically, shift involves segment-level operations across time. The shift operation randomly shifts a given sample x of T time steps to the right by n time steps and concatenates it to the left side of the original data.

[0023] Formally, the proposed time-context data augmentation operation is shown in the following formula, namely:

[0024]

[0025] Among them, x is the sample, and are the corresponding enhanced data respectively.

[0026] Preferably, the signal strength enhancement stage adopts a Drift operation, which is used to construct corresponding positive samples by randomly and smoothly deviating the values ​​of the time series from their original values. The specific drift degree is controlled by the maximum drift and the number of drift points. The specific schematic diagram can be seen in Figure 2 The drift operation divides the data into n segments and performs a drift operation on each segment, that is, randomly enhancing or weakening the signal strength of the original data.

[0027] Preferably, the optimization task module mainly utilizes the dual-view mask prediction contrast loss task.

[0028] The dual-view mask prediction contrast loss task ensures that the encoder can effectively learn cross-dimensional dependencies by masking at the dimensionality level in the encoder spatial representation.

[0029] Preferably, the dual-view mask prediction contrast loss task In this framework, the present invention adopts a dual-view mask prediction task based on temporal contrast, removing the mask contrast learning, and the details can be found in Figure 3 .

[0030] Specifically, for sample x, two types of enhancements ct1 and ct2 are given respectively through dual-view enhancement, and they are regarded as corresponding positive sample pairs. Then, through the dual-view mask prediction task, the information is completed, that is, the latent vector representation ct with missing information after masking is used to restore the original data step by time. The present invention measures the degree of similarity of restoration by contrast loss. In a specific implementation, the present invention randomly extracts batch samples of size N, and performs data enhancement on these data to obtain 2N enhanced data. For the construction of positive sample pairs, the present invention considers that the two views obtained after data enhancement of the same data are logarithmically transformed to obtain h ′ t The representation h of the original data at this time step t is a positive sample, and the other original data are negative samples.

[0031] In the prediction task, the present invention adopts a time contrast module. It uses the contrast loss of the autoregressive model at each time step to extract data features in the latent space. Given the latent representation z, the autoregressive model f θ The time series data z in the latent space is summarized as a context vector ct = f(z), ct∈R×h, where h is the hidden dimension of the autoregressive model f. Then the context feature vector ct is used to restore the original data time by time. In order to better perform data recovery, the present invention uses a logarithmic bilinear model. This model retains the input x t The mutual information between ct, f(x t ,ct)=exp(W t (ct) T z t ), where W t Map ct back to a linear function of the same dimension as z. The specific loss function is defined as follows:

[0032]

[0033] Where T′ represents the matrix transpose, T is the time step, t is the time step, represents the context vector obtained by weak transformation, represents the context vector obtained by strong transformation, represents the latent space representation of the original data at time step t, N t Represents the set of all samples at each time step, and n is the remaining samples at the same time step.

[0034] Compared with the prior art, the present invention improves the prediction and regression accuracy in scenarios with a small amount of labeled data. That is, it uses a contrastive learning pre-training training architecture and introduces two-stage data enhancement of time series and signal intensity to learn more detailed data features of the original data, thereby improving the prediction and regression accuracy. For example, in the industrial chemical fiber data scenario, the CV value (strip unevenness) prediction accuracy is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings in the embodiments are briefly introduced below.

[0036] Figure 1 This is an algorithm architecture diagram of a semi-supervised temporal regression system based on latent space dimensional mask reconstruction according to a preferred embodiment of the present invention.

[0037] Figure 2 A schematic diagram of Drift data enhancement proposed in a preferred embodiment of the present invention.

[0038] Figure 3 A self-prediction task graph based on time series comparison is proposed for a preferred embodiment of the present invention.

[0039] Figure 4 It is a block diagram of a semi-supervised temporal regression system based on latent space dimensional mask reconstruction according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention through specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] The present invention mainly focuses on the performance optimization of semi-supervised time series classification problems, improves the existing time series regression method according to the characteristics of time series data regression tasks, enhances the original data through a strong-weak transformation technology based on time series and signal strength, and adopts a dual-view mask prediction task that is more conducive to downstream regression tasks in the optimization task, so that it can be effectively applied to semi-supervised time series regression problems. It is described in more detail in the form of embodiments.

[0042] A preferred embodiment of the present invention provides a semi-supervised time series regression system based on latent space dimensional mask reconstruction, which is used to effectively apply to the semi-supervised time series regression problem and improve the model performance in scenarios with a small amount of labeled data.

[0043] This embodiment provides a method to improve the model performance in scenarios with a small amount of labeled data, such as Figure 1 , 4 As shown in the figure, it includes the original data enhancement module, the cross-dimensional space encoding module and the optimization task module. The specific introduction of each module is as follows:

[0044] The original data enhancement module performs data enhancement on the original data in terms of timing and signal strength. In this module, the present invention regards the Shift operation as a weak enhancement in terms of timing and the Drift operation as a strong enhancement in terms of signal strength through corresponding data enhancement parameters, thereby achieving the requirements of strong and weak view data enhancement.

[0045] The cross-dimensional spatial encoding module uses the Crossformer encoder to achieve refined representation at the dimensional level and capture the complex dependencies between dimensions. The cross-dimensional spatial encoding module introduces the Crossformer encoder for feature extraction, which overcomes the limitations of the traditional Transformer architecture, can effectively distinguish the potential interdependencies between dimensions, and refine the representation at the dimensional level.

[0046] Optimizing the task module, the present invention removes more contrastive learning losses applied to time series classification tasks and adopts a dual-view mask prediction task to better improve the performance of time series regression tasks.

[0047] In this embodiment, the original data enhancement module deploys the following two stages.

[0048] (1) Timing enhancement stage

[0049] The time series enhancement stage uses a weakly enhanced shift operation to construct positive samples by shifting the original data in the time series arrangement. It is expected that the model can effectively capture the essential characteristics of the original data in terms of time series.

[0050] Specifically, shift involves segment-level operations across time. The shift operation randomly shifts a given sample x of T time steps to the right by n time steps and concatenates it to the left side of the original data.

[0051] Formally, the proposed time-context data augmentation operation is shown in the following formula, namely:

[0052]

[0053] Among them, x is the sample, and are the corresponding enhanced data respectively.

[0054] (2) Signal strength enhancement stage

[0055] The signal strength enhancement stage uses the Drift operation, which is used to construct the corresponding positive samples by randomly and smoothly deviating the values ​​of the time series from their original values. The specific drift degree is controlled by the maximum drift and the number of drift points. The specific schematic diagram can be seen Figure 2 .

[0056] The drift operation divides the data into n segments and performs a drift operation on each segment, that is, randomly enhancing or weakening the signal strength of the original data.

[0057] Cross-dimensional spatial coding encodes different dimensions and different time steps of data accordingly. That is, different dimensions of data at the same time step are encoded differently. Similarly, different time steps of the same dimension are encoded differently. The specific encoding method is represented by a set of learnable parameters.

[0058] In this embodiment, the optimization task module deploys the dual-view mask prediction contrast loss task, the detailed information is as follows: Figure 3 shown.

[0059] Dual-view mask prediction task In this framework, the present invention adopts a dual-view mask prediction task based on temporal comparison, which allows the encoder to effectively learn cross-dimensional dependencies by masking at the dimensional level in the encoder spatial representation. The present invention is different from traditional methods, in which masks are mainly used for data normalization. The dual-view prediction task proposed by the present invention mainly emphasizes masking at the dimensional level in the encoder spatial representation, thereby improving the encoder's ability to learn interdependencies between data dimensions.

[0060] Specifically, for sample x, two types of enhancements ct1 and ct2 are given respectively through dual-view enhancement, and they are regarded as corresponding positive sample pairs. Then, through the dual-view mask prediction task, the information is completed, that is, the latent vector representation ct with missing information after masking is used to restore the original data step by time. The present invention measures the degree of similarity of restoration by contrast loss. In a specific implementation, the present invention randomly extracts batch samples of size N, and performs data enhancement on these data to obtain 2N enhanced data. For the construction of positive sample pairs, the present invention considers that the two views obtained after data enhancement of the same data are logarithmically transformed to obtain h ′ t The representation h of the original data at this time step t is a positive sample, and the other original data are negative samples.

[0061] In the prediction task, the present invention adopts a time contrast module. It uses the contrast loss of the autoregressive model at each time step to extract data features in the latent space. Given the latent representation z, the autoregressive model f θ The time series data z in the latent space is summarized as a context vector ct = f(z), ct∈R×h, where h is the hidden dimension of the autoregressive model f. Then the context feature vector ct is used to restore the original data time by time. In order to better perform data recovery, the present invention uses a logarithmic bilinear model. This model retains the input x t The mutual information between ct, f(x t ,ct)=exp(W t (ct) T z t ), where W t Map ct back to a linear function of the same dimension as z. The specific loss function is defined as follows:

[0062]

[0063] Where T′ represents the matrix transpose, T is the time step, t is the time step, represents the context vector obtained by weak transformation, represents the context vector obtained by strong transformation, represents the latent space representation of the original data at time step t, N t Represents the set of all samples at each time step, and n is the remaining samples at the same time step.

[0064] In summary, the present invention proposes an innovative semi-supervised time series learning method, a semi-supervised time series regression system based on latent space dimensional mask reconstruction, which aims to solve the problem of time series analysis in the case of scarce labeled data. The present invention uses the original data data enhancement module to enhance the original data in terms of time series and signal strength, effectively learns the time series dependency and cross-dimensional dependency of the data, and generates stable enhanced view encoding; the cross-dimensional space encoding module is used to perform refined representation at the dimensional level and capture the complex dependencies between dimensions; the task module is optimized to better improve the performance of the time series regression task, and further enhance the ability to capture time series dependency and cross-dimensional dependency.

[0065] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope of the present invention.

Claims

1. A semi-supervised temporal regression system based on latent space dimensional mask reconstruction, characterized by Includes the following modules: The original data enhancement module is used to enhance the original data in terms of time sequence and signal strength to generate enhanced view coding; The cross-dimensional spatial encoding module uses the Crossformer encoder to achieve refined representation at the dimension level and capture the dependencies between dimensions; Optimize the task module and use the dual-view mask prediction contrast loss task to improve the performance of the time series regression task.

2. The semi-supervised temporal regression system based on latent space dimensional mask reconstruction as claimed in claim 1, characterized in that: The original data enhancement module uses the Shift operation as a weak enhancement in timing and the Drift operation as a strong enhancement in signal strength through the data enhancement parameters.

3. The semi-supervised temporal regression system based on latent space dimensional mask reconstruction as claimed in claim 2, characterized in that: In the process of time series enhancement, a Shift operation with weak time series enhancement is used to construct positive samples by shifting the original data in the time series arrangement. It is expected that the model can effectively capture the essential characteristics of the original data in terms of time series.

4. The semi-supervised temporal regression system based on latent space dimensional mask reconstruction as claimed in claim 2, characterized in that: During the signal strength enhancement process, the Drift operation is used to construct corresponding positive samples by randomly and smoothly deviating the values ​​of the time series from the original values.

5. The semi-supervised temporal regression system based on latent space dimensional mask reconstruction as claimed in claim 3, characterized in that: The shift operation involves segment-level operations across time; a combined operation of time-context data enhancement is used, as shown in the following formula: Among them, x is the sample, and are the corresponding enhanced data respectively.

6. The semi-supervised temporal regression system based on latent space dimensional mask reconstruction as claimed in claim 1, characterized in that: The dual-view mask prediction contrast loss task ensures that the encoder can effectively learn cross-dimensional dependencies by masking at the dimension level in the Crossformer encoder spatial representation.

7. The semi-supervised temporal regression system based on latent space dimensional mask reconstruction as claimed in claim 6, characterized in that: The dual-view mask prediction contrast loss task adopts a dual-view mask prediction task based on temporal comparison.

8. The semi-supervised temporal regression system based on latent space dimension mask reconstruction as claimed in claim 7, characterized in that: For a sample x, two types of enhancements ct1 and ct2 are given respectively through dual-view enhancement, and they are regarded as corresponding positive sample pairs; the information is completed through the dual-view mask prediction contrast loss task, that is, the latent vector ct with missing information after masking is used to represent the original data step by time; the degree of similarity of restoration is measured by contrast loss; batch samples of size N are randomly selected, and data enhancement is performed on these sample data to obtain 2N enhanced data; for the construction of positive sample pairs, it is considered that the same data is obtained after data enhancement and logarithmic transformation h′ t The representation h of the original data at this time step t is a positive sample, and is a negative sample with other original data; In the prediction task, a time contrast module is used to extract data features in the latent space using the contrast loss of the autoregressive model at each time step; Given the latent representation z, the autoregressive model f θ The time series data z in the latent space is summarized as a context vector ct = f(z), ct∈R×h, where R is represented by a real number and h is the hidden dimension of the autoregressive model f; the context feature vector ct is used to restore the original data time step by time.

9. The semi-supervised temporal regression system based on latent space dimensional mask reconstruction as claimed in claim 8, characterized in that: The original data is restored as follows: Use the logarithmic bilinear model, which will retain the input x t The mutual information between ct, f(x t ,ct)=exp(W t (ct) T z t ), z t represents the representation of the data at time step t in the latent space; W t Map ct back to a linear function of the same dimension as z; the specific loss function is defined as follows: Where T′ represents the matrix transpose, T is the time step, t is the time step, represents the context vector obtained by weak transformation, represents the context vector obtained by strong transformation, represents the latent space representation of the original data at time step t, N t Represents the set of all samples at each time step, and n is the remaining samples at the same time step.

Citation Information

Cited By

  • Hydroelectric power generation power prediction method and system

    CN122508109A

  • A method and system for hydroelectric power prediction

    CN122508109B