Self-supervised training generation method suitable for multivariable time series prediction pre-training model

Abstract

The invention discloses a self-supervised training generation method suitable for a multivariable time series prediction pre-training model, and the method comprises the steps: screening a pre-training data set based on morphological similarity; the method comprises the following steps: comparing a representative time sequence of a data set to be screened with a time sequence of a target power source load (wind power, photovoltaic and load) in periodicity, non-negative fluctuation, stability and other dimensions, and selecting a data set with similar features with the time sequence of the target source load as a pre-training data set; then, a dynamic channel mask rate mechanism is introduced in a multivariable self-supervised training stage, only part of channels are randomly selected to participate in loss calculation in the initial stage of training, the proportion of the participating channels is gradually increased according to preset scheduling along with the iteration process, and finally transition to full-channel joint prediction is conducted. Compared with the prior art, the method can improve the matching degree of the pre-training data and the target task, improves the convergence stability and prediction precision of the multivariate pre-training model, and reduces the development and application cost of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of self-supervised training generation method suitable for multivariate time series prediction pre-training model, belong to electric power system time series prediction field. BACKGROUND

[0002] The source and load of power system (specifically including wind power, photovoltaic and load) prediction is the key technology to ensure the stable operation and optimal scheduling of power grid. Traditional small parameter prediction model is often limited to specific prediction scene, and the adaptability to distribution change and data scarcity scene is insufficient. Researching time series pre-training model suitable for multiple scenes, multiple time and multiple spatial scales has become a new research hotspot. Time series prediction pre-training model is pre-trained on massive and diverse time series data sets through self-supervised pre-training, learns the extraction method of general time series pattern and feature, and can be transferred to downstream tasks through fine-tuning. This "pre-training-fine-tuning" paradigm reflects the strong generalization ability of multiple scenes, can standardize the generation process of prediction model, and can effectively reduce the economic cost of model development and maintenance and large-scale application. But the existing pre-training model has two problems in the training process: the training data set lacks screening, and the model training convergence effect is not good in multivariate prediction task. SUMMARY

[0003] The present application aims at the deficiencies of the prior art, and provides a kind of self-supervised training generation method suitable for multivariate time series prediction pre-training model, specifically includes two main steps of time series pre-training data set screening based on morphological similarity and dynamic channel mask rate self-supervised training for multivariate time series prediction task, which are executed in sequence, can effectively improve the generation efficiency and quality of pre-training model.

[0004] The present application adopts the following technical scheme: A self-supervised training generation method for pre-trained models of multivariate time series prediction is proposed. First, a time series pre-training dataset selection based on morphological similarity is performed: representative source-load sequences are randomly selected from power system source-load data (including wind power, photovoltaic, and load) as target power source-load time series; representative time series are randomly selected from open-source time series datasets (such as LOTSA, UTSD, etc.), and the morphological similarity between the selected representative time series and the target power source-load time series is comprehensively scored based on three dimensions: periodicity, non-negative volatility, and stationarity. This score is then used for sorting and selection, improving the relevance and effectiveness between the pre-training data and the target task from the source. Second, self-supervised training with dynamic channel masking rate for multivariate time series prediction tasks is performed: the pre-trained model with an encoder-decoder structure is self-supervised using the multivariate time series samples obtained in the previous step. By gradually transitioning from "few channels, easy task" to "multiple channels, full task" with dynamic masking rate design, the difficulty of early multivariate prediction training is significantly reduced without changing the model structure, improving the stability of loss descent and training convergence efficiency. This invention organically combines dataset selection with dynamic mask training to form a complete multivariate time series pre-training generation method. It can be embedded into existing pre-training-fine-tuning development processes and can be extended to other multivariate time series prediction scenarios, with good versatility and engineering application value.

[0005] In the above technical solution, the selection of time series pre-training datasets based on morphological similarity specifically involves comparing the differences between the representative time series of the dataset to be screened and the target power source load time series in three dimensions: periodicity, non-negative volatility, and stationarity. Datasets with similar morphology are then selected as pre-training datasets. The sequences that best represent the main morphological patterns in the original dataset are called representative sequences, which can be obtained using the K-Means clustering method.

[0006] Furthermore, let the set of open-source time series datasets to be screened be: Among them, the Datasets It must contain at least one scalar time series. For each dataset Randomly select a representative time series , denoted as: in Representative time series Length, For time indexing.

[0007] Randomly select representative source-load sequences from power system source-load (including wind power, photovoltaic and load) data as target power source-load time series : And take it as the benchmark for shape similarity calculation. Under the unified index system, the periodic index, non-negative volatility index and stationarity index are calculated And , the shape difference is constructed, and various difference quantities are linearly combined into shape comprehensive difference degree according to the preset weight, and the shape similarity score is obtained through a monotonically decreasing mapping function , so as to sort and select all open source time series data sets, realize the automatic selection of time series basic model pre-training data set for power system source-load prediction task.

[0008] Specifically, in order to depict the statistical characteristics and periodic characteristics of time series, the present application first calculates the sample mean And sample variance Of the representative time series , which is specifically expressed as: , The sample mean and sample variance of the target power source-load time series Are respectively denoted as And , and the definition method is the same as And . In order to measure the periodicity of time series under a certain lag, the present application introduces the sample autocorrelation index. For the representative time series , the sample autocorrelation At lag Is defined as: The sample autocorrelation Of the target power source-load time series The definition method is the same as , only , , Are replaced by , , . Let the day cycle (Day) of the power source-load correspond to the lag step Under the unified sampling scale, the week cycle (Week) corresponds to the lag step , then the present application defines the first Data set (I.e. ) and Difference vector in the periodic dimension for: Used to characterize the differences between open-source datasets and target power source loads in typical daily and weekly autocorrelation structures.

[0009] Regarding nonnegative volatility and stationarity, this invention constructs key indicators reflecting "nonnegative behavior" and "trend superimposed with short-term volatility behavior." Specifically, for time series... Define the proportion of nonnegative samples Zero value ratio and coefficient of variation , respectively represented as , , in This is an indicator function; it takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. In the denominator This is a constant set to prevent the denominator from being zero. Target power source load time series. Corresponding indicators , and The definition method is related to the proportion of non-negative samples respectively. Zero value ratio With coefficient of variation The same. This invention will and Non-negative volatility difference vector Defined as: To characterize stationarity, this invention uses the ratio of the variance of the sequence to be screened to the variance of the first-difference sequence to represent the relative strength of the trend and short-term fluctuations. For representative time scalar sequences... The first-order difference is defined as: Its sample mean is defined as: The first-order difference variance is defined as: Combining equation (1), the variance of the representative time series of the dataset to be screened is taken. The stability coefficient Defined as: Target power source-load time series stationarity coefficient Definition method and Maintain consistency. Define a stationarity difference scalar. for: To synthesize the three types of differences mentioned above—periodicity, non-negative volatility, and stationarity—this invention introduces a periodic weight vector. Non-negative volatility weight vector and stationarity weight scalar : , , Each component is a non-negative real number and satisfies . No. Datasets Morphological diversity Defined as: Meanwhile, to enhance interpretability and facilitate sorting, this invention employs a monotonically decreasing mapping function to integrate morphological differences. Mapped to morphological similarity score The conversion formula is: .when hour, This indicates that the shapes are completely identical; with Increase Monotonically decreasing. Ultimately, this invention is based on morphological similarity scores. For all open-source time series datasets Sort the datasets and select the top-ranked datasets based on a preset similarity threshold or by ranking them by score to form a pre-training dataset set. ,Will It is applied to the first stage of training of the pre-trained model, thereby enabling the selection and interpretable use of open-source time series datasets that are "similar to the power source and load patterns".

[0010] Furthermore, the dynamic channel mask rate self-supervised training for multivariate time series prediction tasks specifically involves: introducing a dynamic channel mask rate mechanism to perform self-supervised training on the pre-trained model; the dynamic channel mask rate mechanism specifically involves: in the early stage of training the pre-trained model, only a portion of the channels are randomly selected to participate in the loss calculation, and as the iteration progresses, the proportion of participating channels is gradually increased according to a preset mask rate scheduling strategy, eventually transitioning to joint prediction of all channels.

[0011] Furthermore, to address the issues of poor convergence efficiency, large loss fluctuations, and difficulty in stably learning effective features from multivariate time series prediction pre-trained models during training, this invention proposes a dynamic channel mask rate self-supervised training method for multivariate time series prediction tasks, specifically for time series pre-trained models based on encoder-decoder or Transformer network structures. The multivariate time series samples used in the pre-training stage are obtained through the aforementioned steps, and the model's input within a given time window is a three-dimensional tensor. The output prediction target is a three-dimensional tensor. ,in, This indicates the batch size of data input to the model in a single instance. Indicates the length of the input history sequence. Indicates the predicted length. This represents the number of variables or channels. After calculation by the embedding layer and several encoder and decoder layers, the model reaches its final value in the [number of]th [stage]. The predicted output tensor generated during the training iteration is denoted as: The corresponding real target tensor is denoted as: superscript The corresponding number is The next iteration. Let the loss function calculated point by point be... (For example, when the loss function is set to squared error,) When set to absolute error, Traditional multivariate training processes typically use the average loss across all channels and all time steps. As a training objective: , , in and They represent the first time. During the nth iteration, the 1st Individual samples, prediction time steps ,aisle The predicted and actual values ​​are compared. Because all channels participate in the loss calculation simultaneously in a multivariate scenario, the prediction task in a single iteration is quite difficult. The model struggles to stably capture the joint patterns between multiple channels in the early stages of training, resulting in high training loss. It often exhibits a trend of large fluctuations and difficulty in convergence, affecting the model's convergence speed and prediction performance.

[0012] To overcome the above shortcomings, this invention proposes a loss construction method that incorporates dynamic channel mask rate in self-supervised multivariate prediction training. In the... During the next training iteration, the mask rate is introduced. This represents the proportion of channels participating in the loss calculation in this iteration. Let the minimum mask rate be... Maximum mask rate Then for the first k The next iteration can define the total number of channels participating in the loss calculation. for: in This means rounding down to the nearest integer. satisfy In the first In each iteration, for each batch of sample index From the channel index set The total number of elements randomly selected without replacement with equal probability is 1. A subset of , denoted as : , in Indicates the size of the set. Based on sets Construct a 3D channel mask tensor The elements inside this tensor are defined as follows: That is, in the same batch of samples Within, all predicted time steps Sharing the same set of selected channels To ensure loss normalization and comparability, this invention defines the total number of valid elements participating in loss calculation as follows: : Based on this, construct the first... The dynamic channel mask loss function for the next iteration is: in accordance with The calculated loss is used for backpropagation and model parameter updates. This is different from traditional methods that average the loss across all channels. Loss of the present invention Only randomly selected in the current iteration The error is accumulated on one channel, and the error of the other channels is not included in the current gradient calculation and parameter update, although the model still outputs the predicted value. This effectively reduces the task difficulty of a single iteration in the early stage of training, and enables the model to learn stable time-dependent structures and step-related features on some channels first.

[0013] To gradually improve the model's learning ability for multivariate joint prediction tasks, this invention focuses on mask rate. The learning mechanism is dynamically adjusted according to a cosine annealing scheduling strategy during training, constructing a channel-level learning mechanism that progresses from simple to complex. Let the predetermined total number of training epochs be... The number of sample batches in a single training round is The total number of iterations during the entire training process is defined as follows: During training, for the first The first round of training Each batch, its global iterative index This can be expressed as: , , Among them, the global iterative index Should meet The normalized training progress is defined as follows: , During the training process, The mask rate monotonically increases from 0 to 1 from the start to the end of training. During this process, to increase the mask rate from a simple prediction task with a low initial value to a complex task of full-channel prediction, this invention employs a monotonically increasing mask rate scheduling strategy based on cosine annealing. The update strategy is defined as follows: in, The initial mask rate is used to control the number of prediction channels involved in calculating the loss during the early stages of training. It is usually set to 0.1 or 0.05 depending on the complexity of the prediction task. The maximum mask ratio indicates that the prediction results of all channels are included in the loss calculation during the later stages of training. It can also be modified as needed. Other values ​​within the range, but must strictly satisfy... This formula ensures that the mask ratio is close to the target value in the early stages of training. The number of channels involved in the loss calculation increases monotonically to 1 according to the cosine curve as the iteration progresses. It grows monotonically and smoothly from a small initial value to the total number of channels. In the early stages of training, the model only needs to be trained on a small number of channels, thus significantly alleviating the problems of overly complex loss surfaces and large gradient fluctuations in multivariate prediction tasks. As the training progresses, the mask rate is gradually increased using a cosine annealing strategy, and the model gradually transitions to training on all channels. The complete task of jointly predicting from multiple channels.

[0014] Throughout the training process, this invention consistently employs a defined channel mask loss. As an optimization basis for backpropagation, combined with the pre-training dataset obtained by the aforementioned morphological similarity screening, the training convergence efficiency and final prediction accuracy of the multivariate time series pre-training model can be significantly improved.

[0015] The beneficial effects of this invention are: This invention constructs a morphological similarity evaluation method based on periodicity, non-negative volatility, and stationarity characteristics. It filters open-source time-series datasets, prioritizing data with similar power source / load morphology for pre-training. This avoids interference from irrelevant or excessively different datasets, improving the effectiveness of data utilization and adaptability to power scenarios during pre-training. It also addresses the lack of filtering in pre-training datasets, enhancing the matching degree between data and tasks. By introducing a dynamic channel mask rate, loss is calculated only on a subset of channels in the early stages of training, reducing the task difficulty of a single iteration and allowing the model to stabilize learning time and cross-channel features on a small number of channels initially. As training progresses, the number of channels involved in loss calculation is gradually increased, ultimately achieving joint prediction for all channels. This process effectively alleviates the problem of drastic loss fluctuations and difficulty in convergence during multivariate prediction training, improving the prediction accuracy of pre-trained models in multivariate source / load prediction tasks such as wind power, photovoltaics, and load, improving training convergence characteristics in multivariate prediction scenarios, and enhancing model stability and accuracy. Compared with existing technologies, this invention can improve the matching degree between pre-training data and target tasks, improve the convergence stability and prediction accuracy of multivariate pre-trained models, and reduce the cost of model development and application. Attached Figure Description

[0016] Figure 1 This is a flowchart of a time series pre-training dataset selection method based on morphological similarity; Figure 2 This is a flowchart of the mask rate update process for the dynamic channel mask rate self-supervised training method. Detailed Implementation

[0017] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0018] A self-supervised training generation method for pre-trained models of multivariate time series prediction is proposed, which includes the following steps: Step 1: Filter the time series pre-training dataset based on morphological similarity (e.g., Figure 1 The specific method is as follows: Let the set of open-source time series datasets to be filtered be: Among them, the Datasets It must contain at least one scalar time series. This invention applies to each dataset. Randomly select a representative time series , denoted as: in Representative time series Length, For time indexing.

[0019] Representative source load sequences are randomly selected from power system source load data (including wind power, photovoltaic, and loads) to serve as the target power source load time series. : Will As a benchmark for morphological similarity calculation.

[0020] To characterize the statistical properties and periodicity of time series, we first analyze each representative time series. Calculate the sample mean With sample variance Specifically: , Target power source-load time series The sample mean and sample variance are denoted as . and Its definition method is the same as and The same. To measure the periodicity of a time series under a specific lag, a sample autocorrelation index is introduced. For representative time series... In lag Autocorrelation of samples at location Defined as: Target power source-load time series Sample autocorrelation Definition method and The same, only during the calculation process , , Replace with , , Let the lag step corresponding to the daily cycle (Day) of the power source load under a uniform sampling scale be... The lag step corresponding to the week is Then the present invention defines the first Datasets (Right now )and Difference vector in the periodic dimension for: The Used to characterize the differences between open-source datasets and target power source loads in typical daily and weekly autocorrelation structures.

[0021] Regarding nonnegative volatility and stationarity, this invention constructs a key indicator reflecting "nonnegative behavior" and "trend superimposed on short-term volatility behavior." For time series... Non-negative sample proportion Zero value ratio and coefficient of variation They are defined as follows: , , in This is an indicator function; it takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. In the denominator This is a constant set to prevent the denominator from being zero. Target power source load time series. Corresponding indicators , and The definition methods are respectively with , and The same. This invention will and Non-negative volatility difference vector Defined as: To characterize stationarity, this invention uses the ratio of the variance of the sequence to be screened to the variance of the first-difference sequence to represent the relative strength of the trend and short-term fluctuations. For representative time scalar sequences... The first-order difference is defined as: Its sample mean is defined as: The first-order difference variance is defined as: Combining equation (1), the variance of the representative time series of the dataset to be screened is taken. The stability coefficient Defined as: Target power source-load time series stationarity coefficient Definition method and Maintain consistency. Define the stationarity difference. for: To synthesize the three types of differences mentioned above—periodicity, non-negative volatility, and stationarity—this invention introduces a periodic weight vector. Non-negative volatility weight vector and stationarity weight scalar : , , Each component is a non-negative real number and satisfies . No. Datasets Morphological diversity Defined as: .

[0022] And a monotonically decreasing mapping function is used to integrate the morphological differences. Mapped to morphological similarity score The conversion formula is: .when hour, This indicates that the shapes are completely identical; with Increase It decreases monotonically.

[0023] Finally, scores are based on morphological similarity. For all open-source time series datasets Sort the datasets and select the top-ranked datasets based on a preset similarity threshold or by ranking them by score to form a pre-training dataset set. ,Will It is applied to the first stage of training of the pre-trained model, thereby enabling the selection and interpretable use of open-source time series datasets that are "similar to the power source and load patterns".

[0024] Step Two: To address the issues of poor convergence efficiency, large loss fluctuations, and difficulty in stably learning effective features from multivariate time series prediction pre-trained models during training, this invention proposes a dynamic channel mask rate self-supervised training method for multivariate time series prediction tasks, specifically for time series pre-trained models based on encoder-decoder or Transformer network structures. The multivariate time series samples used in the pre-training stage are obtained through the screening process described in Step One. The model's input within a given time window is a three-dimensional tensor. The output prediction target is a three-dimensional tensor. ,in, This indicates the batch size of data input to the model in a single instance. Indicates the length of the input history sequence. Indicates the predicted length. This represents the number of variables or channels. After calculation by the embedding layer and several encoder and decoder layers, the model reaches its final value in the [number of]th [stage]. The predicted output tensor generated during the training iteration is denoted as: The corresponding real target tensor is denoted as: superscript The corresponding number is The next iteration. Let the loss function calculated point by point be... (For example, when the loss function is set to squared error,) When set to absolute error, Traditional multivariate training processes typically use the average loss across all channels and all time steps. As a training objective: , , in and They represent the first time. During the nth iteration, the 1st Individual samples, prediction time steps ,aisle The predicted and actual values ​​are compared. Because all channels participate in the loss calculation simultaneously in a multivariate scenario, the prediction task in a single iteration is quite difficult. The model struggles to stably capture the joint patterns between multiple channels in the early stages of training, resulting in high training loss. It often exhibits a trend of large fluctuations and difficulty in convergence, affecting the model's convergence speed and prediction performance.

[0025] To overcome the above shortcomings, this invention proposes a loss construction method that incorporates dynamic channel mask rate in self-supervised multivariate prediction training. In the... During the next training iteration, a mask rate scalar is introduced. This represents the proportion of channels participating in the loss calculation in this iteration. Let the minimum mask rate be... Maximum mask rate Then for the first k The total number of channels that can be defined to participate in the loss calculation in this iteration is: in This means rounding down to the nearest integer. satisfy In the first In each iteration, for each batch of sample index From the channel index set A random sample of a number with equal probability and without replacement is drawn from a sample of a total of 1. A subset of , denoted as : , in Indicates the size of the set. Based on sets This invention constructs a three-dimensional channel mask tensor. The elements inside this tensor are defined as follows: That is, in the same batch of samples Within, all predicted time steps Sharing the same set of selected channels To ensure loss normalization and comparability, this invention defines the total number of valid elements participating in loss calculation as follows: : Based on this, construct the first... The dynamic channel mask loss function for the next iteration is: in accordance with The calculated loss results are then used for backpropagation and model parameter updates.

[0026] like Figure 2 As shown, in order to gradually improve the model's learning ability for multivariate joint prediction tasks, this invention increases the mask rate. A channel-level learning mechanism, ranging from simple to complex, is constructed through dynamic adjustments based on cosine annealing during training. Let the predetermined total number of training epochs be... The number of sample batches in a single training round is The total number of iterations during the entire training process is defined as follows: During training, for the first The first round of training Each batch, its global iterative index This can be expressed as: , , Among them, the global iterative index Should meet The normalized training progress is defined as follows: , During the training process, The mask rate monotonically increases from 0 to 1 from the start to the end of training. During this process, to increase the mask rate from a simple prediction task with a low initial value to a complex task of full-channel prediction, this invention employs a monotonically increasing mask rate scheduling strategy based on cosine annealing. The update strategy is defined as follows: in, The initial mask rate is used to control the number of prediction channels involved in calculating the loss during the early stages of training. It is usually set to 0.1 or 0.05 depending on the complexity of the prediction task. The maximum mask ratio indicates that the prediction results of all channels are included in the loss calculation during the later stages of training. It can also be modified as needed. Other values ​​within the range, but must strictly satisfy... This formula allows control over the number of channels involved in loss calculation. It grows monotonically and smoothly from a small initial value to the total number of channels. .

[0027] Throughout the training process, this invention consistently employs a defined channel mask loss. As an optimization basis for backpropagation, combined with the pre-training dataset obtained by the aforementioned morphological similarity screening, the training convergence efficiency and final prediction accuracy of the multivariate time series pre-training model can be significantly improved.

Claims

1. A self-supervised training generation method suitable for pre-trained models of multivariate time series prediction, characterized in that, Includes the following steps: Step 1) Screen the pre-training dataset based on morphological similarity. By comparing the differences between the representative time series of the dataset to be screened and the target power source load time series in three dimensions of periodicity, non-negative fluctuation and stationarity, select the dataset with similar morphology as the pre-training dataset. Step 2) Based on the pre-trained dataset, a dynamic channel masking rate mechanism is introduced to perform self-supervised training on the pre-trained model. The dynamic channel masking rate mechanism is as follows: in the early stage of training the pre-trained model, only a portion of the channels are randomly selected to participate in the loss calculation. As the iteration progresses, the proportion of participating channels is gradually increased according to the preset masking rate scheduling strategy, and finally transitions to joint prediction of all channels.

2. The self-supervised training generation method for multivariate time series prediction pre-trained models according to claim 1, characterized in that, Step 1) specifically includes the following steps: 1.1) Feature Construction: From each open-source time series dataset to be screened Select representative time series And calculate the sample mean. With sample variance According to sample autocorrelation : Construct the target power source load time series based on power system source load data. And calculate the sample mean, sample variance and sample autocorrelation. ; Lag step on a given day cycle With the lag step of the cycle Under the condition of constructing a periodic difference vector : Calculate the representative time series for each dataset. Non-negative sample ratio Zero value ratio With coefficient of variation : , , ,in, This is an indicator function; it takes the value 1 when the condition within the parentheses is true, and 0 otherwise. Similarly, it constructs the non-negative sample proportion of the target power source-load time series. Zero value ratio With coefficient of variation And construct a nonnegative volatility difference vector. : Then, calculate the first-order difference variance and stationarity coefficient of the representative time series and the target power source load time series for each dataset, respectively. The stationarity coefficient of the representative time series... The calculation formula is: , and The variance and first-order difference variance of the representative time series of the dataset to be screened are respectively, and a stationarity difference scalar is constructed. : ; 1.2) Morphological similarity calculation: Define a periodic weight vector that is non-negative and sums to 1. Non-negative volatility weight vector and stationarity weight scalar and weighted calculation of the dataset Morphological diversity : Then, the overall morphological differences are considered. Mapped to morphological similarity score Scoring based on morphological similarity For all datasets to be filtered Sort the datasets and select the top-ranked datasets based on a preset similarity threshold or by ranking them by score to form a pre-training dataset set. .

3. The self-supervised training generation method for multivariate time series prediction pre-trained models according to claim 1, characterized in that, In step 2), the training method for the kth iteration is as follows: 2.1) in the In each training iteration, a multivariate prediction model using a Transformer architecture outputs a prediction tensor. ,in, This indicates the batch size of data input to the model in a single instance. Indicates the length of the input history sequence. Indicates the predicted length. Indicates the number of variables or channels; according to and its corresponding real target tensor Define the pointwise loss function Total average loss of batch ; 2.2) Set the mask rate for the current iteration And calculate the number of channels involved in the loss calculation. : ; 2.3) Sample index for each batch In the channel index set The total number of randomly selected elements is subset of And further construct the channel mask tensor , The internal elements are defined as: ; 2.4) Calculate the total number of valid elements. : and construct the first The dynamic channel mask loss function of the next iteration : ,in accordance with The calculated loss results are then used for backpropagation and model parameter updates. Wherein, the mask rate The k-th iteration of training is updated according to the cosine annealing scheduling strategy.

4. The self-supervised training generation method for multivariate time series prediction pre-trained models according to claim 3, characterized in that, The mask rate The schedule is updated according to the cosine annealing scheduling strategy with each training iteration k. The specific steps include: (1) Pre-set the total number of training epochs. and the number of sample batches in a single training round And calculate the total number of iterations. ; (2) Define the normalized training progress And define the cosine annealing scheduling strategy: , , To ensure that the mask rate is close to the target value in the early stages of training. , and as the iteration progresses, it monotonically increases to 1 according to the cosine curve.

Images