Multi-variate missing time series end-to-end prediction method and device, electronic equipment

Abstract

The application discloses a kind of multi-element missing time series data end-to-end prediction method, comprising: obtaining the multi-element missing time series data sample of historical time period with prediction label and corresponding data missing matrix;Build neural network model based on Transformer;Multi-element missing time series data sample is normalized to instance, and is divided into non-overlapping time block, obtain time block information matrix and time block missing matrix;Manual information masking is carried out to time block information matrix and time block missing matrix, and pre-training neural network model is obtained by the way of reconstructing masking information for self-supervised pre-training;Joint cross-prediction module is constructed, and time block information matrix and time block missing matrix are input into pre-training neural network model and cross-prediction module in turn to obtain prediction result, and supervised fine-tuning is carried out according to label to obtain multi-element missing time series prediction model;Input multi-element missing time series data is predicted using multi-element missing time series prediction model to obtain time series prediction result.

Description

Technical Field

[0001] This application relates to the field of multivariate time series prediction, specifically to a multivariate missing time series end-to-end prediction method, apparatus, and electronic equipment. Background Technology

[0002] Multivariate time series data is a primary data type in real-world scenarios, and its prediction capabilities have proven highly practical, such as in stock price forecasting and weather forecasting. Data gaps are a frequent problem in multivariate time series data analysis. Due to various factors such as data acquisition equipment malfunctions, unstable system environments, or privacy concerns, certain features / values ​​in a multivariate time series may be unobservable. Data gaps directly lead to a decline in the quality of multivariate time series data, impacting the effectiveness of subsequent downstream tasks. Therefore, it is necessary to propose corresponding methods for analyzing multivariate time series data with missing data.

[0003] Existing research on multivariate time series prediction focuses on the problem of predicting complete multivariate time series data, generating predictions by learning historical patterns from complete datasets, but lacks end-to-end methods for predicting missing multivariate time series data. An intuitive approach to predicting missing multivariate time series data is to use a multivariate time series imputation model to estimate missing values, and then use a prediction model to predict future time series. However, these methods still have limitations: (1) The need to impute before prediction limits efficiency. (2) Errors exist during the imputation process, which may be amplified during prediction, leading to poor accuracy. (3) Most current methods ignore the dependencies between features, which are crucial in multivariate time series tasks. Summary of the Invention

[0004] To overcome the above technical problems, embodiments of this application provide a multivariate missing time series end-to-end prediction method, apparatus, and electronic device to improve prediction accuracy and efficiency.

[0005] According to a first aspect of the embodiments of this application, an end-to-end prediction method for multivariate missing time series data is provided, characterized in that it includes:

[0006] Obtain multivariate missing time series data samples with predicted labels for historical time periods and their corresponding missing data matrices;

[0007] Construct a neural network model based on Transformer;

[0008] After performing instance normalization on the multivariate missing time series data samples according to their corresponding missing data matrices, the time block information matrix and its time block missing matrix are obtained by dividing the data into non-overlapping time blocks.

[0009] The time block information matrix and its missing time block matrix are manually masked. The masked time block information matrix and its missing time block matrix are then concatenated and input into the Transformer-based neural network model to obtain a temporal representation. Reconstruction and prediction are performed through a linear layer. Self-supervised pre-training is performed by reconstructing the masking information to obtain the pre-trained neural network model.

[0010] A joint cross-prediction module is constructed. The spliced ​​time block information matrix and its time block missing matrix are input into the pre-trained neural network model to obtain the time series representation. Then, the time series representation is input into the joint cross-prediction module to obtain the prediction result. The prediction result is denormalized. The pre-trained neural network model is fine-tuned by minimizing the mean square error of the time series prediction result and the future time series label to obtain a multivariate missing time series prediction model.

[0011] The spliced ​​time block information matrix and its time block missing matrix are input into the multivariate missing time series prediction model to predict the multivariate missing time series data. After obtaining the results, they are inversely normalized to obtain the final time series future prediction results.

[0012] According to a second aspect of the embodiments of this application, an end-to-end prediction apparatus for multivariate missing time series data is provided, comprising:

[0013] The acquisition module is used to acquire multivariate missing time series data samples with predicted labels for historical time periods and their corresponding missing data matrices.

[0014] Modules for building Transformer-based neural network models;

[0015] The data processing module is used to perform instance normalization on the multivariate missing time series data samples according to their corresponding missing data matrices, and then obtain the time block information matrix and its time block missing matrix by dividing the data into non-overlapping time blocks.

[0016] The pre-training module is used to manually mask the time block information matrix and its missing time block matrix, concatenate the masked time block information matrix and its missing time block matrix, input the concatenated matrix into the Transformer-based neural network model to obtain the temporal representation, and perform reconstruction prediction through a linear layer. Self-supervised pre-training is performed by reconstructing the masking information to obtain the pre-trained neural network model.

[0017] The fine-tuning module is used to construct the joint cross-prediction module. The concatenated time block information matrix and its time block missing matrix are input into the pre-trained neural network model to obtain the temporal representation. Then, the temporal representation is input into the joint cross-prediction module to obtain the prediction result. The prediction result is denormalized, and the pre-trained neural network model is fine-tuned by minimizing the mean square error of the temporal prediction result and the future time series label to obtain the multivariate missing temporal prediction model.

[0018] The prediction module is used to input the spliced ​​time block information matrix and its time block missing matrix into the multivariate missing time series prediction model to predict the multivariate missing time series data. After obtaining the result, it performs inverse normalization to obtain the final time series future prediction result.

[0019] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0020] One or more processors;

[0021] Memory, used to store one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0023] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0024] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0025] Compared with the prior art, the improvements of this invention are as follows: This method adopts a Transformer-based architecture to perform end-to-end prediction of multivariate missing time series data, overcoming the problem of low prediction efficiency of missing time series data; it learns the distribution of missing states through pre-training and adapts to downstream prediction tasks through fine-tuning, overcoming the problem of inaccurate prediction of missing time series data. The prediction accuracy of this method is 20% higher than the current algorithm of first completing and then predicting, and it has higher prediction efficiency.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] Figure 1This is a flowchart illustrating an end-to-end prediction method for multivariate missing time series data according to an exemplary embodiment.

[0029] Figure 2 This is a block diagram illustrating the training of a multivariate missing time series prediction model according to an exemplary embodiment.

[0030] Figure 3 This is a block diagram illustrating an end-to-end prediction method apparatus for multivariate missing time series data according to an exemplary embodiment. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] Existing research on multivariate time series prediction focuses on the problem of predicting complete multivariate time series data, generating predictions by learning historical patterns from complete datasets, but lacks end-to-end methods for predicting missing multivariate time series data. An intuitive approach to predicting missing multivariate time series data is to use a multivariate time series imputation model to estimate missing values, and then use a prediction model to predict future time series. However, these methods still have limitations: (1) The need to impute before prediction limits efficiency. (2) Errors exist during the imputation process, which may be amplified during prediction, leading to poor accuracy. (3) Most current methods ignore the dependencies between features, which are crucial in multivariate time series tasks.

[0033] To overcome the above technical problems, embodiments of this application provide a multivariate missing time series end-to-end prediction method, apparatus, and electronic device to improve prediction accuracy and efficiency.

[0034] The multivariate missing time series end-to-end prediction method provided in this application is applicable to time series prediction in almost all scenarios, such as weather condition prediction and air quality prediction, and provides a good solution for time series prediction problems under data missing conditions. For ease of description of the time series data prediction method of this application, weather prediction is used as an example in this embodiment.

[0035] Figure 1 This is a flowchart illustrating an end-to-end prediction method for multivariate missing time series data according to an exemplary embodiment, such as... Figure 1 As shown, the method may include the following steps:

[0036] S1: Obtain multivariate missing time series data samples with predicted labels for historical time periods and their corresponding missing data matrices; this step may include the following sub-steps:

[0037] S11: Acquire multivariate missing time-series data for historical periods collected by the sensor;

[0038] Specifically, you can select data from the past month, which can have many fields. For example, weather data for a month can include multiple feature fields such as date, temperature, precipitation, and relative humidity.

[0039] S12: Construct multivariate missing time series data samples and corresponding prediction labels for the multivariate missing time series data using a sliding window method;

[0040] Specifically, based on the missing multivariate time-series data of the historical time period, multiple samples are constructed using a sliding time window method. That is, a time-series dataset is constructed by sliding a time window of length L with a step size of 1. Where N is the number of samples. Where x i =(x i1 ,…,x iL ), where d represents the number of data fields and L represents the time window length. For each sample X, there exists a corresponding future time series prediction label. in Where T is the preset length of the future time to be predicted.

[0041] S13: Based on the missing data situation, obtain the data missing matrix corresponding to the multivariate missing time series data sample.

[0042] Specifically, due to sampler malfunctions or environmental instability, samples may have missing data. Therefore, it is necessary to obtain the missing data matrix M = (m1,…,m) corresponding to sample X based on the missing data situation. i ,…,m d )∈{0,1} d×L , where m i =(m i1 ,…,m iL ), m ij =1 indicates that x ij A value of 0 indicates an observable state, while a value of 0 indicates a missing state.

[0043] S2: Construct a neural network model based on Transformer;

[0044] Specifically, a Transformer-based neural network model is constructed, which consists of an embedding layer and multiple Transformer blocks. The embedding layer consists of a learnable linear mapping matrix and a position encoding layer, while each Transformer block consists of a multi-head attention mechanism, a feedforward neural network, residual connections, and layer normalization.

[0045] S3: After performing instance normalization on the multivariate missing time series data samples according to their corresponding missing data matrices, the time block information matrix and its time block missing matrix are obtained by dividing the data into non-overlapping time blocks; this step may include the following sub-steps:

[0046] S31: Based on the data missing matrix corresponding to the multivariate missing time series sample, perform instance normalization on all observable time series data in the multivariate missing time series sample using the Z-Score method to obtain the normalized multivariate missing time series sample.

[0047] Specifically, for each sample Where d represents the number of features in the time series, and L represents the length of the time series, the instances will be normalized using Z-Score standardization. That is, for each univariate time series x i First, calculate sample μ. i and sample variance σ i 2 As shown below:

[0048]

[0049] Normalization is achieved by subtracting the mean and dividing by the standard deviation:

[0050]

[0051] Finally, we obtained the normalized multivariate missing time series samples. This application employs instance normalization followed by inverse normalization after obtaining the prediction results to mitigate the distribution offset effect in incomplete multivariate time series data, thereby further improving prediction accuracy.

[0052] S32: Divide the normalized multivariate missing time series samples into multiple non-overlapping time blocks to obtain the divided time block information matrix and its time block missing matrix.

[0053] Specifically, the normalized multivariate missing time series samples are divided by non-overlapping partitioning. It is divided into K non-overlapping temporal subsequence blocks of length P, where Obtain the time block information matrix after partitioning and the corresponding missing time block matrix in and This application adopts a time-series block method to overcome the problem of excessive model time complexity caused by excessively long time series, thereby improving prediction efficiency.

[0054] S4: Manually mask the time block information matrix and its missing time block matrix, concatenate the masked time block information matrix and its missing time block matrix, input the concatenated matrix into the Transformer-based neural network model to obtain temporal representation, and perform reconstruction prediction through a linear layer. Self-supervised pre-training is then performed using the reconstructed masking information to obtain the pre-trained neural network model. This step may include the following sub-steps:

[0055] S41: Perform information masking at the time block granularity on the non-overlapping time blocks with a masking rate of γ to generate a masking matrix;

[0056] Specifically, a masking matrix is ​​generated for the normalized multivariate missing time series samples using the γ masking ratio. in A vector consisting entirely of zeros indicates that the information in that time block is masked; otherwise, a vector consisting entirely of one indicates that the information in that time block is visible. This application overcomes the problem of models predicting masking information from adjacent temporal information by using a block-level masking method, thereby enhancing the robustness of the pre-trained model.

[0057] S42: Fill the masked information in the time block information matrix with 0, and set the masked time block status in the time block missing matrix to missing, to obtain the masked time block information matrix and the missing matrix.

[0058] Specifically, for the masked time block, all its information will be manually filled with 0 to obtain the masked time block information matrix, and the masked time block status in the time block missing matrix will be set to missing to obtain the masked missing matrix.

[0059] S43: After concatenating the masked time block information matrix and the missing matrix, the time sequence representation is obtained by inputting the Transformer-based neural network model in an independent manner between feature channels.

[0060] Specifically, the masked time block information matrix and the missing information matrix are concatenated and used as input to obtain the input for each time block. Where || represents a join operation.

[0061] First, the feature variables are independent of each other through the embedding layer in the Transformer-based neural network model, including the use of a learnable linear matrix W.p The input is mapped to an embedding vector of dimension D, and a positional encoding matrix W is added. pos To identify temporal location information, an embedding matrix is ​​obtained. The specific process is as follows:

[0062]

[0063] The embedding matrix is ​​then input into a neural network consisting of R-layer Transformer blocks to represent the time series. Specifically, for each embedding matrix... Each layer will output an intermediate representation. in In each layer, the model first captures the dependencies between different locations through a multi-head attention mechanism and layer normalization, and then outputs the results through fully connected layers and residual connected layers, as follows:

[0064]

[0065]

[0066] Where MHA, LN, and MLP represent multi-head attention operation, layer normalization operation, and fully connected layer operation, respectively. This leads to the temporal representation. This application concatenates time information and missing information and inputs them into the model for representation, enabling the model to perceive the missing state information of the time series, thereby improving the prediction accuracy.

[0067] S44: The time series representation is mapped to time series values ​​through a linear layer, and the model is trained by reconstructing observable masking information to obtain a pre-trained neural network model.

[0068] Specifically, during the pre-training phase, a linear layer is used to project the representation, resulting in... Here, LL represents a linear operation. The model is trained using batch gradient descent with a reconstruction error loss function, as follows:

[0069]

[0070] in This represents the training sample set. A pre-trained model is obtained through continuous optimization and training. This application pre-trains the model by reconstructing the masked visible information, learning the distribution of missing states, and mitigating the problem of limited time-series data samples to some extent, thus enabling the model to obtain robust initial parameters.

[0071] S5: Construct a joint cross-prediction module. Input the concatenated time block information matrix and its time block missing matrix into the pre-trained neural network model to obtain a temporal representation. Then, input the temporal representation into the joint cross-prediction module to obtain the prediction result. Inverse normalize the prediction result and fine-tune the pre-trained neural network model by minimizing the mean squared error of the temporal prediction result and the future time series labels to obtain a multivariate missing temporal prediction model. This step may include the following sub-steps:

[0072] S51: Construct a joint cross-prediction module, which consists of an expanded layer and multiple linear layers, and can map each time series representation into a prediction matrix for all time series;

[0073] S52: After concatenating the time block information matrix and the missing matrix, the time sequence representation is obtained by inputting them into the pre-trained neural network model in an independent manner between feature channels.

[0074] Specifically, the time block information matrix and the missing matrix are concatenated to obtain... As input, the time-series representation is obtained by inputting the pre-trained neural network model in a feature-independent manner.

[0075] S53: Input the time series representation into the joint cross-prediction module to obtain the cross-prediction matrix, and combine all cross-prediction results in a weighted manner to obtain the joint prediction result;

[0076] Specifically, the representation is input into the joint cross-prediction module, where it is processed through an expansion layer and multiple linear layers. Further mapping yields the cross-prediction matrix. in Let represent the cross-prediction matrix of the i-th time series with respect to all time series. The specific process is as follows:

[0077]

[0078] Where LL represents the linear layer and FL represents the expanded layer. The module will comprehensively consider all cross-predictions to obtain the joint prediction result. Specifically It will be calculated using the following formula:

[0079]

[0080] Where β is a hyperparameter used to adjust the impact of other time series on the current time series prediction. This represents the cross-prediction vector of the j-th time series to the i-th time series. This application utilizes all observable information to predict the future by using the cross-prediction module to predict the representation, while also considering the correlation between features, thus further improving prediction accuracy.

[0081] S54: After denormalizing the joint prediction results, the final prediction result is obtained. The pre-trained neural network model is fine-tuned by minimizing the mean square error between the final prediction result and the future time series label to obtain the multivariate missing time series prediction model.

[0082] Specifically, the final prediction result is obtained by inverse normalizing the joint prediction result. The pre-trained neural network model is then fine-tuned by minimizing the mean square error between the final prediction result and the future time series labels. The specific objectives are as follows:

[0083]

[0084] Among them l e This represents the mean square error function. Representing time series x i The future true values ​​are obtained. Finally, a multivariate missing time series model is obtained. This application fine-tunes the model by minimizing the mean squared error of the prediction results and future time series labels, enabling the model to adapt to downstream tasks and thus improving its performance on prediction tasks.

[0085] S6: Input the spliced ​​time block information matrix and its time block missing matrix into the multivariate missing time series prediction model to predict the multivariate missing time series data. After obtaining the result, perform inverse normalization to obtain the final time series future prediction result.

[0086] Specifically, the time block information matrix and the missing matrix are concatenated to obtain... The prediction results are obtained by inputting the multivariate missing time series prediction model, and the results are inversely normalized to obtain the final time series future prediction results.

[0087] Corresponding to the aforementioned embodiments of the multivariate missing time series data end-to-end prediction method, this application also provides embodiments of the multivariate missing time series data end-to-end prediction apparatus.

[0088] Figure 3 This is a block diagram of an end-to-end prediction apparatus for multivariate missing time-series data according to an exemplary embodiment. (Refer to...) Figure 3 The device includes:

[0089] Module 1 is used to acquire multivariate missing time series data samples with predicted labels for historical time periods and their corresponding missing data matrices.

[0090] Module 2 is used to build a Transformer-based neural network model;

[0091] Data processing module 3 is used to perform instance normalization processing on the multivariate missing time series data samples according to their corresponding data missing matrix, and then obtain the time block information matrix and its time block missing matrix by dividing the data into non-overlapping time blocks.

[0092] The pre-training module 4 is used to manually mask the time block information matrix and its time block missing matrix, concatenate the masked time block information matrix and its time block missing matrix, input the concatenated matrix into the Transformer-based neural network model to obtain the temporal representation, and perform reconstruction prediction through a linear layer. Self-supervised pre-training is performed by reconstructing the masking information to obtain the pre-trained neural network model.

[0093] The fine-tuning module 5 is used to construct a joint cross-prediction module. It inputs the spliced ​​time block information matrix and its time block missing matrix into the pre-trained neural network model to obtain a time series representation. Then, it inputs the time series representation into the joint cross-prediction module to obtain a prediction result. The prediction result is denormalized, and the pre-trained neural network model is fine-tuned by minimizing the mean square error of the time series prediction result and the future time series label to obtain a multivariate missing time series prediction model.

[0094] Prediction module 6 is used to input the spliced ​​time block information matrix and its time block missing matrix into the multivariate missing time series prediction model to predict the multivariate missing time series data, and then perform inverse normalization to obtain the final time series future prediction result.

[0095] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0096] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0097] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multivariate missing time series data end-to-end prediction method as described above.

[0098] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the end-to-end prediction method for multivariate missing time-series data as described above.

[0099] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0100] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An end-to-end prediction method for multivariate missing time series data, characterized in that, The method is used for weather forecasting and includes: Obtain multivariate missing time series data samples with prediction labels for historical time periods and their corresponding missing data matrices; the multivariate missing time series data samples include four feature fields: date, temperature, precipitation, and relative humidity over a period of time; Construct a neural network model based on Transformer; After performing instance normalization on the multivariate missing time series data samples according to their corresponding missing data matrices, the time block information matrix and its time block missing matrix are obtained by dividing the data into non-overlapping time blocks. The time block information matrix and its missing time block matrix are manually masked. The masked time block information matrix and its missing time block matrix are then concatenated and input into the Transformer-based neural network model to obtain a temporal representation. Reconstruction and prediction are performed through a linear layer. Self-supervised pre-training is performed by reconstructing the masking information to obtain the pre-trained neural network model. A joint cross-prediction module is constructed. The spliced ​​time block information matrix and its time block missing matrix are input into the pre-trained neural network model to obtain the time series representation. Then, the time series representation is input into the joint cross-prediction module to obtain the prediction result. The prediction result is denormalized. The pre-trained neural network model is fine-tuned by minimizing the mean square error of the time series prediction result and the future time series label to obtain a multivariate missing time series prediction model. The spliced ​​time block information matrix and its time block missing matrix are input into the multivariate missing time series prediction model to predict the multivariate missing time series data. After obtaining the results, they are inversely normalized to obtain the final time series future prediction results.

2. The method according to claim 1, characterized in that, Obtain multivariate missing time series data samples with predicted labels for historical time periods and their corresponding missing data matrices, including: Acquire multivariate missing time-series data from historical periods collected by sensors; Multivariate missing time series data samples and corresponding predicted labels are constructed using a sliding window method for the aforementioned multivariate missing time series data; Based on the missing data, obtain the data missing matrix corresponding to the multivariate missing time series data sample.

3. The method according to claim 1, characterized in that, Constructing a Transformer-based neural network model includes: A Transformer-based neural network model is constructed, which consists of an embedding layer and multiple Transformer blocks. The embedding layer consists of a learnable linear mapping matrix and a positional encoding layer, while each Transformer block consists of a multi-head attention mechanism, a feedforward neural network, residual connections, and layer normalization.

4. The method according to claim 1, characterized in that, After performing instance normalization on the multivariate missing time series data samples according to their corresponding missing data matrices, time block information matrices and their time block missing matrices are obtained by dividing the data into non-overlapping time blocks, including: Based on the data missing matrix corresponding to the multivariate missing time series sample, the Z-Score method is used to perform instance normalization on all observable time series data in the multivariate missing time series sample to obtain the normalized multivariate missing time series sample. The normalized multivariate missing time series samples are divided into multiple non-overlapping time blocks to obtain the divided time block information matrix and its time block missing matrix.

5. The method according to claim 1, characterized in that, The time block information matrix and its missing time block matrix are manually masked. The masked time block information matrix and its missing time block matrix are then concatenated and input into the Transformer-based neural network model to obtain temporal representations. Reconstruction and prediction are performed through linear layers. Self-supervised pre-training is then performed using the reconstructed masking information to obtain the pre-trained neural network model, including: Information masking at the time block granularity is performed on the non-overlapping time blocks using a masking rate to generate a masking matrix; Fill the masked information in the time block information matrix with 0, and set the masked time block status in the time block missing matrix to missing, to obtain the masked time block information matrix and the missing matrix. After concatenating the masked time block information matrix and the missing matrix, the time sequence representation is obtained by inputting them into the Transformer-based neural network model in an independent manner between feature channels. The time series representation is mapped to time series values ​​through a linear layer, and the model is trained by reconstructing observable masking information to obtain a pre-trained neural network model.

6. The method according to claim 1, characterized in that, A joint cross-prediction module is constructed. The concatenated time block information matrix and its missing time block matrix are input into the pre-trained neural network model to obtain a temporal representation. Then, the temporal representation is input into the joint cross-prediction module to obtain prediction results. The prediction results are inversely normalized. The pre-trained neural network model is fine-tuned by minimizing the mean squared error of the temporal prediction results and future time series labels to obtain a multivariate missing time series prediction model, including: A joint cross-prediction module is constructed, which consists of an expanded layer and multiple linear layers, and can map each time series representation into a prediction matrix for all time series. After concatenating the time block information matrix and the missing matrix, the time sequence representation is obtained by inputting them into the pre-trained neural network model in an independent manner between feature channels. The time series representation is input into the joint cross-prediction module to obtain the cross-prediction matrix, and the joint prediction result is obtained by combining all cross-prediction results in a weighted manner. The joint prediction results are denormalized to obtain the final prediction result. The pre-trained neural network model is fine-tuned by minimizing the mean square error between the final prediction result and the future time series labels to obtain the multivariate missing time series prediction model.

7. An end-to-end prediction device for multivariate missing time series data, characterized in that, This device is used for weather forecasting and includes: The acquisition module is used to acquire multivariate missing time series data samples with prediction labels for historical time periods and their corresponding missing data matrices; the multivariate missing time series data samples include four feature fields: date, temperature, precipitation and relative humidity over a period of time. Modules for building Transformer-based neural network models; The data processing module is used to perform instance normalization on the multivariate missing time series data samples according to their corresponding missing data matrices, and then obtain the time block information matrix and its time block missing matrix by dividing the data into non-overlapping time blocks. The pre-training module is used to manually mask the time block information matrix and its missing time block matrix, concatenate the masked time block information matrix and its missing time block matrix, input the concatenated matrix into the Transformer-based neural network model to obtain the temporal representation, and perform reconstruction prediction through a linear layer. Self-supervised pre-training is performed by reconstructing the masking information to obtain the pre-trained neural network model. The fine-tuning module is used to construct the joint cross-prediction module. The concatenated time block information matrix and its time block missing matrix are input into the pre-trained neural network model to obtain the temporal representation. Then, the temporal representation is input into the joint cross-prediction module to obtain the prediction result. The prediction result is denormalized, and the pre-trained neural network model is fine-tuned by minimizing the mean square error of the temporal prediction result and the future time series label to obtain the multivariate missing temporal prediction model. The prediction module is used to input the spliced ​​time block information matrix and its time block missing matrix into the multivariate missing time series prediction model to predict the multivariate missing time series data. After obtaining the result, it performs inverse normalization to obtain the final time series future prediction result.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-6.

Images