Learning method for incomplete multivariate time series prediction

Abstract

The embodiment of the invention provides a method for training a student model, and the method comprises the steps: obtaining a first training set which comprises a plurality of original samples which are not subjected to missing processing and corresponding labels; obtaining a teacher model trained by using the original sample and the label, wherein the teacher model comprises a feature extractor used for extracting teacher characterization from the original sample and a regression layer used for obtaining a teacher prediction result according to the teacher characterization; processing each original sample by using a plurality of missing rates to obtain corresponding missing samples, and forming a second training set by all the missing samples and the corresponding labels; obtaining a student model, wherein the student model comprises a feature extractor used for extracting student characterization from the missing sample and a regression layer used for obtaining a student prediction result according to the student characterization; and training the student model by using the second training set, the teacher representation and the teacher prediction result, and during training, updating parameters of the student model based on the representation loss, the prediction result loss, the comparison loss and the total loss weighted by the absolute error loss.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, in particular to the field of multivariate time series applications in the field of machine learning, and more particularly to a learning method for incomplete multivariate time series prediction. Background Art

[0002] Multivariate time series is a dataset consisting of two or more related time series. These time series can be different indicators under the same theme, such as multiple related data indicators in the fields of economy, meteorology, and medical care. Multivariate time series forecasting is an important direction in the field of predictive analysis and data science. It involves using multiple time series variables at the same time to predict future values, which can help enterprises and government departments make more accurate decisions with limited information. Compared with univariate time series forecasting, multivariate time series forecasting utilizes the relationship between various variables and can provide more accurate and comprehensive forecast results. However, due to the nonlinearity of time series itself and the complex spatial dependencies between different time series, the accuracy of forecasting has always been a difficult problem in academia and industry.

[0003] Mainstream multivariate time series forecasting techniques can generally be divided into statistical methods, machine learning methods, and deep learning methods. Among them, statistical methods rely on formalizing the time series modeling process into mathematical formulas using prior knowledge. This method is relatively effective when dealing with stationary data with strong regularity; however, for non-stationary data, this method often cannot fully utilize expert experience to analyze the dynamic changes of data. Machine learning methods, such as support vector machines and random forests, use the idea of ​​nonlinear modeling to enhance the analysis of time series historical data and the ability to predict future trends. However, machine learning methods are mainly designed for univariate time series forecasting and fail to effectively analyze the interactions between multiple time series.

[0004] Compared with statistical methods and machine learning methods, deep learning methods have attracted widespread attention due to their excellent nonlinear modeling capabilities and in-depth analysis of spatial relationships. Classic deep learning models, such as recursive neural networks and convolutional neural networks, effectively mine the spatiotemporal dependencies between multivariate time series through sequence modeling and convolutional layer technology, respectively. In addition, with the continuous expansion of computing resources, spatiotemporal graph neural networks and attention-based Transformer models have further enhanced the use of contextual information and the modeling of spatial relationships, making deep learning models widely used.

[0005] Although deep learning models have achieved satisfactory results, the evaluation data sets they use are usually relatively complete and cleaned. However, in real life, multivariate time series prediction tasks usually face an important challenge, namely the problem of missing data, which is usually caused by problems with the data collector. When the data collector is affected by bad weather or natural disasters, depending on the time and location of the natural disaster, the data collector usually fails at different times, resulting in random data loss in both time and space dimensions. In general, the abnormality of the data collector will cause the multivariate time series to be incomplete. The faulty sensor will not be able to output valid values ​​for a period of time or multiple periods, resulting in the destruction of the original time series. Moreover, when some sensors fail, the acquired multivariate time series only contains data collected by some normal data collectors, and the data bits corresponding to the faulty sensors are 0.

[0006] When existing models are applied to datasets with a large data missing rate, the models usually find it difficult to produce satisfactory prediction results. In addition, the data missing rate usually changes over time, and existing work usually requires training multiple models for different missing rates, which limits the robustness of the model.

[0007] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0008] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a learning method for incomplete multivariate time series prediction.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] According to a first aspect of the present invention, a method for training a student model is provided, the method comprising: obtaining a first training set, which comprises a plurality of original samples that have not been processed for missingness and a label corresponding to each original sample, the original sample being a historical observation value, and the label being a true value observed after the corresponding historical observation value; obtaining a teacher model trained using the original samples and the labels, which comprises a feature extractor for extracting teacher representations from the original samples and a regression layer for obtaining teacher prediction results based on the teacher representations; performing missingness processing on each original sample in the first training set using a variety of missing rates to obtain corresponding missing samples, and all missing samples and corresponding labels constitute a second training set; obtaining a student model, which comprises a feature extractor for extracting student representations from missing samples The invention relates to a method for preparing a student model of an image processing unit and a regression layer for obtaining a student prediction result according to the student representation; the student model is trained using the second training set, the teacher representation and the teacher prediction result, wherein during training, the parameters of the student model are updated based on a preset total loss function to obtain a trained student model, and the total loss function includes a weighted sum of the following loss functions: a representation loss function for determining the loss between the teacher representation and the student representation; a prediction result loss function for determining the loss between the teacher prediction result and the student prediction result; a contrast loss function for determining the contrast loss between the student representations of different missing samples corresponding to the same original sample and between the student representations of missing samples corresponding to different original samples; and an absolute error loss function for determining the loss between the student prediction result and the corresponding label.

[0011] In some embodiments of the present invention, the total loss function is:

[0012]

[0013] in, represents the representation loss function, represents the prediction result loss function, represents the contrast loss function, represents the absolute error loss function, α, β, γ, and λ represent the weights corresponding to the representation loss function, prediction result loss function, contrast loss function, and absolute error loss function, respectively.

[0014] In some embodiments of the present invention, the representation loss function is:

[0015]

[0016] Among them, N represents the number of original samples, m represents the number of missing rates, represents the teacher representation of the nth original sample, represents the student representation of the i-th missing sample corresponding to the n-th original sample, and Mean(·) represents the mean of the tensor.

[0017] In some embodiments of the present invention, the prediction result loss function is:

[0018]

[0019] Among them, N represents the number of original samples, m represents the number of missing rates, represents the teacher's prediction result of the nth original sample, represents the student prediction result of the i-th missing sample corresponding to the n-th original sample, and Mean(·) represents the mean of the tensor.

[0020] In some embodiments of the present invention, the contrast loss is determined in the following manner: obtain student representations corresponding to all missing samples, and take the student representations of all missing samples corresponding to each missing rate as a representation sample set; form a sample group with every two representation sample sets, and determine anchor samples, positive samples and negative samples from each sample group, wherein the positive samples and the anchor samples are the student representations corresponding to missing samples from the same original sample but with different missing rates, and the negative samples and the anchor samples are the student representations corresponding to missing samples from different original samples; calculate the sub-loss of each anchor sample based on the positive and negative samples corresponding to the anchor sample; determine the sub-contrast loss of each sample group based on the sub-loss of all samples in each sample group; determine the contrast loss based on the sub-contrast losses of all sample groups based on a preset contrast loss function.

[0021] In some embodiments of the present invention, the contrast loss function is expressed as:

[0022]

[0023] Among them, m represents the number of missing rates, PQ represents the sample group consisting of the Pth representation sample set and the Qth representation sample set, and L PQ represents the sub-contrastive loss of sample group PQ.

[0024] In some embodiments of the present invention, when the student model is a meteorological forecasting model, the historical observation values ​​are a multivariate time series consisting of meteorological data at different locations, and the labels are true meteorological values ​​observed after the corresponding historical observation values; or, when the student model is a traffic flow forecasting model, the historical observation values ​​are a multivariate time series consisting of traffic flow data at multiple intersections, and the labels are true values ​​of traffic flow data observed after the corresponding historical observation values.

[0025] According to the second method of the present invention, a method for realizing prediction using a trained student model obtained by the method described in the first aspect of the present invention is provided, which comprises: obtaining historical observation values; inputting the historical observation values ​​into the trained student model for prediction to obtain a prediction result.

[0026] Compared with the prior art, the advantages of the present invention are:

[0027] By setting multiple missing rates, the original samples are missing, so as to simulate the missing data collected in the actual application scenario, so that the total loss function is obtained by weighted summing the representation loss function determined by the representation of the original samples and the representation of the missing samples, the prediction result loss function determined by the prediction result of the original samples and the student prediction result of the missing samples, the contrast loss function determined by the representation of the missing samples, and the absolute error loss function determined by the prediction result of the missing samples and the corresponding label. It can not only reduce the difference between the representation of the missing samples and the representation of the original samples, and the difference between the prediction result of the missing samples and the prediction result of the original samples, but also shorten the distance between the missing samples with different missing rates belonging to the same original sample, and increase the distance between the missing samples that do not belong to the same original sample; the student model updated according to the total loss function has better generalization ability, whether it is the prediction of complete data or the prediction of missing data, its prediction results are more accurate, especially for the prediction of missing data, it has good prediction accuracy for data under different missing rates, and its prediction performance is more stable, which solves the problem that the existing model cannot predict data with multiple missing rates and the prediction accuracy is not high, so that the model obtained by this method has more practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0029] Figure 1 The figure is a flowchart of a method for training a student model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] As mentioned in the background technology section, although the existing methods have achieved satisfactory results, they are aimed at relatively complete and cleaned data. In real life, due to the impact of environmental factors on data collectors, data is randomly missing in time and space dimensions. When the existing model is applied to a data set with a large data missing rate, the model is usually difficult to produce satisfactory prediction results. In addition, the data missing rate is usually changing over time, and the existing work usually requires training multiple models for different missing rates, that is, setting multiple missing rates, collecting samples for each missing rate and training a model separately, and then first detecting the missing rate of the input data, and then inputting it into the model corresponding to the missing rate for prediction, which limits the robustness of the model.

[0032] In order to solve the above problems, the present invention proposes a method for training a student model, which obtains a teacher model by using raw data and labels that have not been processed for missing data, and uses the teacher model to extract the raw data that has not been processed for missing data to obtain a teacher representation and a teacher prediction result; uses the student model to extract missing samples to obtain a student representation and a student prediction result, wherein the missing samples are obtained by processing the original samples for missing data using multiple missing rates to simulate the missing data collected in actual application scenarios; the teacher representation and the teacher prediction result extracted by the teacher model are passed to the student model as knowledge, and the teacher representation, the teacher prediction result, the student representation and the student prediction result are used to train the student model. The student model is trained. During the training, the parameters of the student model are updated by the total loss function obtained by weighted summing up the representation loss function determined by the teacher representation and the student representation, the prediction result loss function determined by the teacher's prediction result and the student's prediction result, the contrast loss function determined by the student representation, and the absolute error loss function determined by the student's prediction result and the corresponding label. In this way, the trained student model can not only have high prediction accuracy and robustness for non-missing data, but also have high prediction accuracy and robustness for missing data, which solves the problem that the existing model cannot predict data with various missing rates and has low prediction accuracy, making the model obtained by this method more valuable for practical application.

[0033] In order to better understand the present invention, the method proposed in the present invention is applied to the field of traffic flow prediction. Taking traffic prediction as an example, the technical solution of the present invention is described in detail from four aspects: training samples, model structure, training process and application scenarios.

[0034] 1. Training Samples

[0035] Before training the student model, samples need to be organized. According to one embodiment of the present invention, a first training set is obtained, which includes a plurality of original samples that have not been processed for missing values ​​and a label corresponding to each original sample, the original sample is a historical observation value, and the label is a true value observed after the corresponding historical observation value, wherein the historical observation value is a multivariate time series. Schematically, the historical observation value is a multivariate time series composed of traffic flow data of multiple intersections, and the label is the true value of the traffic flow data observed after the corresponding historical observation value, wherein the traffic flow data includes: traffic flow, speed data, occupancy, congestion data and / or vehicle type. For example: the original sample in the first training set is X={x1,x2,x3,x4}, and the corresponding label is Y={y1,y2,y3,y4}, the original sample of an intersection that has not been processed for missing values The corresponding label is the true value observed after the original, for example: etc., among which a i It can be expressed as the traffic flow at the i-th moment, b i It can be expressed as the speed data at the i-th moment, c i It can be expressed as the occupancy rate at the i-th moment. The above is only an example, and the present invention is not limited to this.

[0036] According to an embodiment of the present invention, a plurality of missing rates are used to perform missing processing on each original sample in the first training set to obtain corresponding missing samples, and all missing samples and corresponding labels form a second training set. Multiple missing rates are set to perform missing processing on each original sample in the first training set to simulate missing data collected in actual application scenarios. In an exemplary manner, for an original sample that has not been missing processed, The original sample is processed using three missing rates (25%, 50% and 75%) to obtain the missing samples corresponding to the original sample: and Therefore, the missing samples in the second training set are original samples with three missing rates. The above are only examples, and the present invention is not limited thereto.

[0037] According to an embodiment of the present invention, the original sample can be obtained from an existing traffic flow dataset (e.g., PEMS04 dataset). A sliding window (size N+M) is used to determine the original sample and the corresponding label from the existing dataset, and the historical observations at N moments are used to predict the observations at one moment (M=1) or multiple moments (M≥2) in the future. Schematically, a multivariate time series in the existing dataset is Set the size of the sliding window to 8+1, and use the sliding window to intercept X1 to obtain the original sample: The corresponding label is Or, the original sample is: The corresponding label is That is, use the historical observations at 8 moments to predict the observations at the next moment; set the size of the sliding window to 4+4, use the sliding window to intercept X1, and get the original sample as: The corresponding label is Or, the original sample is: The corresponding label is That is, the historical observation values ​​at four moments are used to predict the observation values ​​at the next four moments. The above is only an example, and it should be understood that those skilled in the art can modify the size of the sliding window to obtain more embodiments.

[0038] 2. Model Structure

[0039] The present invention provides a teacher model and a student model, wherein the teacher model and the student model are traffic flow prediction models for predicting traffic flow, wherein the teacher model comprises a feature extractor for extracting a traffic flow teacher representation from original samples that have not been processed for missingness, and a regression layer for obtaining a traffic flow teacher prediction result based on the traffic flow teacher representation; the student model comprises a feature extractor for extracting a traffic flow student representation from missing samples, and a regression layer for obtaining a traffic flow student prediction result based on the traffic flow student representation.

[0040] According to one embodiment of the present invention, the feature extraction layer for extracting traffic flow representation can be composed of an embedding layer and an L-layer MLP with residual connection, wherein the embedding layer is used to convert traffic flow data into a high-dimensional hidden representation, and the L-layer MLP is used to mine the high-dimensional hidden representation to obtain the traffic flow representation.

[0041] According to an example of the present invention, for a traffic flow data x, an embedding layer is used to transform it to obtain a high-dimensional hidden representation H = PC (x), where FC (·) represents a fully connected layer; L layers of MLP mine H to obtain a representation H of the sample l+1 =FC(Relu(FC(H l )))+H l , where H l represents the traffic flow representation extracted by the first layer of MLP, Relu(·) represents the activation function, and finally the regression layer is used to represent H L Make a prediction and get the traffic flow prediction result Y=FC(H L ).

[0042] 3. Training Process

[0043] Based on the training samples organized as above, the parameters of the student model are updated using the total loss function to obtain a traffic flow prediction model for practical application scenarios.

[0044] According to one embodiment of the present invention, the student model is trained using the second training set, the traffic flow teacher representation and the traffic flow teacher prediction results, wherein during training, the parameters of the student model are updated based on a preset total loss function to obtain a trained student model, and the total loss function is determined according to the weighted sum of the following losses: the representation loss function, the prediction result loss function, the contrast loss function and the absolute error loss function; wherein: the representation loss function is used to determine the loss between the traffic flow teacher representation and the traffic flow student representation; the prediction result loss function is used to determine the loss between the traffic flow teacher prediction results and the traffic flow student prediction results; the contrast loss function is used to determine the contrast loss between the traffic flow student representations of different missing samples corresponding to the same original sample and between the traffic flow student representations of missing samples corresponding to different original samples; the absolute error loss function is used to determine the loss between the traffic flow student prediction results and the corresponding labels.

[0045] According to one embodiment of the present invention, the total loss function can be expressed as:

[0046]

[0047] in, represents the representation loss function, represents the prediction result loss function, represents the contrast loss function, represents the absolute error loss function, α, β, γ, and λ represent the weights corresponding to the representation loss function, prediction result loss function, contrast loss function, and absolute error loss function, respectively.

[0048] In order to enable the student model to have good prediction accuracy for both missing data and non-missing data, the present invention proposes a representation loss function determined according to the sample representation and a prediction result loss function determined according to the sample prediction result. According to one embodiment of the present invention, the representation loss function is determined according to the sub-representation loss function of all original samples, and the sub-representation loss function of each original sample is determined according to the traffic flow teacher representation of each original sample and the traffic flow student representation of the missing sample corresponding to the original sample. The sub-representation loss function of each original sample can be expressed as:

[0049]

[0050] Among them, m represents the number of missing rates, represents the traffic flow teacher representation of the nth original sample, represents the student representation of the traffic flow of the i-th missing sample corresponding to the n-th original sample, and Mean(·) represents the mean of the tensor. Schematically, the representation loss function can be expressed as: Wherein, N represents the number of original samples. More specifically, the representation loss function can be expressed as:

[0051]

[0052] Among them, N represents the number of original samples, m represents the number of missing rates, represents the traffic flow teacher representation of the nth original sample, represents the student representation of the traffic flow of the i-th missing sample corresponding to the n-th original sample, and Mean(·) represents the mean of the tensor. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the student model is trained by using the representation of the data without missing processing and the representation of the data after missing processing, so as to reduce the difference between the representations generated by the student model and the teacher model, so that the representation of the missing samples obtained by the student model is closer to the representation of the complete data, thereby improving the prediction ability of the student model.

[0053] According to an embodiment of the present invention, the prediction result loss is determined by the sub-prediction result loss of all original samples. The sub-prediction result loss of each original sample is determined according to the traffic flow teacher prediction result of each original sample and the traffic flow student prediction result of the missing sample corresponding to the original sample. The sub-prediction result loss function of each original sample can be expressed as:

[0054]

[0055] Among them, m represents the number of missing rates, represents the traffic flow teacher prediction result of the nth original sample, represents the student prediction result of the traffic flow of the i-th missing sample corresponding to the n-th original sample, and Mean(·) represents the mean of the tensor. Schematically, the prediction result loss function can be expressed as: More specifically, the prediction result loss function can be expressed as:

[0056]

[0057] Among them, N represents the number of original samples, m represents the number of missing rates, represents the traffic flow teacher prediction result of the nth original sample, The traffic flow student prediction result of the i-th missing sample corresponding to the n-th original sample is represented. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the student model is trained by using the prediction results of the data without missing processing and the prediction results of the data after missing processing, so as to reduce the difference between the prediction results generated by the student model and the teacher model, so that the prediction results of the student model for the missing samples are closer to the prediction results of the complete data, thereby improving the prediction ability of the student model.

[0058] Considering that the missing rate of historical observations in real scenarios is not fixed, in order to enhance the student model's ability to accurately predict data under different missing rates, the present invention proposes a contrast loss function. According to one embodiment of the present invention, the contrast loss function is determined in the following manner: obtaining the student representations of the traffic flow corresponding to all missing samples, and taking the student representations of the traffic flow of all missing samples corresponding to each missing rate as a representation sample set; forming a sample group with every two representation sample sets, and determining anchor samples, positive samples and negative samples from each sample group, wherein the positive sample and the anchor sample are the student representations of the traffic flow corresponding to the missing samples from the same original sample but with different missing rates, and the negative sample and the anchor sample are the student representations of the traffic flow corresponding to the missing samples from different original samples; calculating the sub-loss of each anchor sample according to the positive sample and negative sample corresponding to the anchor sample; determining the sub-contrast loss of each sample group according to the sub-loss of all samples in each sample group; determining the contrast loss according to the sub-contrast loss of all sample groups. The technical solution of this embodiment can at least achieve the following beneficial technical effects: through contrast loss, the distance between each anchor sample and the student representation of the same original sample as the anchor sample can be shortened, and the distance between each anchor sample and the student representation of the anchor sample that does not belong to the same original sample can be shortened, so that the student model trained according to contrast loss can have good prediction accuracy for data under different missing rates.

[0059] According to one embodiment of the present invention, the sub-loss of each anchor sample can be calculated as follows:

[0060]

[0061] Among them, z p represents the pth anchor sample, z q represents the qth positive sample of the anchor sample, z k represents the kth negative sample of the anchor sample, t represents the temperature parameter, and sim(·) represents the cosine similarity function.

[0062] Schematically, the sub-contrast loss of each sample group is determined according to the sub-loss of all samples in each sample group and can be expressed as:

[0063]

[0064] Among them, PQ represents a sample group consisting of the Pth representation sample set and the Qth representation sample set, and N represents the number of samples in the representation sample set.

[0065] Schematically, the contrast loss function determined according to the sub-contrast losses of all sample groups can be expressed as:

[0066]

[0067] Among them, m represents the number of missing rates, PQ represents the sample group consisting of the Pth representation sample set and the Qth representation sample set, and L PQ represents the sub-contrastive loss of sample group PQ.

[0068] According to another embodiment of the present invention, in order to reduce the complexity of calculation, when calculating the contrast loss, a fully connected layer can be used to reduce the dimension of the traffic flow student representation extracted by the feature extraction layer of the student model to obtain a traffic flow reduced dimension representation that corresponds one-to-one to the traffic flow student representation, and all traffic flow reduced dimension representations corresponding to each missing rate are used as a representation sample set; every two representation sample sets are combined into a sample group, an anchor sample is determined from each sample group, and the traffic flow reduced dimension representation in the sample group that belongs to the same original sample as the anchor sample is used as a positive sample, and the traffic flow reduced dimension representation that does not belong to the same original sample as the anchor sample is used as a negative sample; the sub-loss of each anchor sample is calculated based on the positive sample and negative sample of the anchor sample; the sub-contrast loss of each sample group is determined based on the sub-loss of all samples in each sample group; the contrast loss is determined based on the sub-contrast loss of all sample groups. Schematically, in the sub-loss of each anchor sample given above, Z p , Z q 、z k Represents the traffic flow dimensionality reduction representation of the original sample.

[0069] In addition to the above loss function, the present invention also updates the parameters of the model by combining the absolute error loss function determined by the student prediction results and the corresponding labels. According to one embodiment of the present invention, the absolute error loss function can be expressed as:

[0070]

[0071] in, represents the label of the nth original sample, It represents the student prediction result of the traffic flow of the i-th missing sample corresponding to the n-th original sample.

[0072] According to an example of the present invention, Figure 1As shown in FIG. 1 , a schematic diagram of a method for training a student model proposed in the present invention is shown. There are unprocessed original samples x1, x2, x3, and the corresponding labels are y1, y2, y3. The original samples are processed using three missing rates (25%, 50%, and 75%) to obtain the missing samples x1, x2, and x3 corresponding to the original samples. 11 、x 12 、x 13 , x 21 、x 22 、x 23 , x 31 、x 32 、x 33 , the teacher model is used to extract the traffic flow teacher representation of the original sample and the traffic flow teacher prediction results. The traffic flow teacher representation is: The traffic flow teacher prediction results are: The student model is used to extract the traffic flow student representation and traffic flow student prediction results of the missing samples. The traffic flow student representation is: The traffic flow student prediction results are: According to the student representation of traffic flow, the representation sample set can be obtained as follows: The representation loss is calculated based on the traffic flow teacher representation and the traffic flow student representation. The prediction loss is calculated based on the traffic flow teacher prediction results and the traffic flow student prediction results. The contrast loss under different missing rates is calculated based on the traffic flow student representation. The absolute error loss is calculated based on the traffic flow student prediction results and the corresponding labels. The total loss is obtained by weighted summation of the representation loss, prediction result loss, contrast loss and absolute error loss. The parameters of the student model are updated using the total loss. Among them, the representation loss is: The prediction result loss is: The contrast loss is: The absolute error loss is:

[0073] According to one embodiment of the present invention, the teacher model is trained based on the original samples and labels without missing processing. More specifically, the teacher model can be trained using the original data and labels without missing processing, or can be trained using the first training set. The training process is as follows: input the original samples without missing processing into the teacher model to obtain the corresponding prediction results; calculate the absolute error loss based on the prediction results and the corresponding labels, and update the parameters of the teacher model based on the absolute error loss. The absolute error loss can be expressed as:

[0074]

[0075] in, represents the label of the nth original sample, Represents the traffic flow teacher prediction result of the nth original sample.

[0076] IV. Application Scenarios

[0077] In actual application scenarios, historical observation values ​​are obtained, and the historical observation values ​​are input into the trained student model obtained by the above method to obtain corresponding prediction results.

[0078] According to one embodiment of the present invention, historical observation values ​​are obtained, and the historical observation values ​​are multivariate time series composed of traffic flow data at different locations; the historical observation values ​​are input into the trained student model obtained in the above embodiment to obtain traffic flow prediction results.

[0079] It should be understood that the method proposed in the present invention can be applied to the field of weather forecasting in addition to the field of traffic flow forecasting provided in the aforementioned embodiments, and the data in the field of traffic flow forecasting given in the aforementioned embodiments can be replaced with data in the field of weather forecasting to obtain an embodiment of weather forecasting. Among them:

[0080] In the training sample part, the multivariate time series composed of traffic flow data at multiple intersections is replaced by the multivariate time series composed of meteorological data at different locations. Correspondingly, the labels are replaced by the true values ​​of meteorological data observed after the corresponding historical observations, wherein the meteorological data include: temperature, air pressure, humidity, wind speed, wind direction, precipitation, cloud cover and / or visibility. The existing traffic flow dataset (such as PEMS04 dataset) is replaced by the existing meteorological dataset (such as JenaClimate dataset), and the one at an intersection is replaced by the one at a location. i It can be expressed as: The traffic flow at the i-th moment can be replaced by the temperature, b i It can be expressed as: the speed data at the i-th moment can be replaced by air pressure, c i It can be expressed as the occupancy rate at the i-th moment can be replaced by humidity.

[0081] In the model structure part, the traffic flow teacher representation extracted by the teacher model is replaced by the meteorological teacher representation, and the traffic flow teacher prediction results are replaced by the meteorological teacher prediction results; the traffic flow student representation extracted by the student model is replaced by the meteorological student representation, and the traffic flow student prediction results are replaced by the meteorological student prediction results.

[0082] In the training process, the traffic flow prediction model used in actual application scenarios will be replaced by the meteorological prediction model, the traffic flow teacher representation will be replaced by the meteorological teacher representation, the traffic flow teacher prediction results will be replaced by the meteorological teacher prediction results, the traffic flow student representation will be replaced by the meteorological student representation, the traffic flow student prediction results will be replaced by the meteorological student prediction results, and the traffic flow dimensionality reduction representation will be replaced by the meteorological dimensionality reduction representation.

[0083] In the application scenario part, the multivariate time series composed of traffic flow data at multiple intersections is replaced by the multivariate time series composed of meteorological data at different locations, and the traffic flow prediction results are replaced by meteorological prediction results.

[0084] Based on the method for training the student model given in the above embodiment, in order to make the method more applicable, the present invention provides a corresponding training device, the functional interface and details of the device are shown in Table 1:

[0085] Table 1 Functional interfaces of the training device

[0086]

[0087] In order to demonstrate the predictive ability of the student model obtained by a method for training a student model provided by the present invention, a comparative experiment was conducted between the present invention and the existing scheme. In the experiment, the prediction error of the model for existing data sets with different missing rates was tested. The following experimental results are used to demonstrate the technical effect of the present invention.

[0088] The schemes used for comparative experiments include:

[0089] (1) Use the prediction results and labels to train the MLP, and obtain MLP1;

[0090] (2) MLP2 is obtained by training the MLP including the interpolation model using the prediction results and label pairs;

[0091] (3) The MLP is trained using the training method proposed in the present invention to obtain MLP3.

[0092] The data set used in this comparative experiment is the traffic flow data set (PEMS04). The data set is processed with missing rates of 25%, 50%, 75% and 90% to obtain missing samples. The models obtained by the above three schemes are used to predict missing samples with different missing rates. The prediction error results are shown in Table 2:

[0093] Table 2 Prediction error results of the model obtained by the existing scheme for samples with different missing rates

[0094]

[0095] According to Table 2, no matter what kind of missing rate data, the prediction error of the model (MLP3) obtained by the method proposed by the present invention is lower than the prediction error of the model in other schemes, and as the missing rate increases, the prediction error in each scheme increases accordingly, but the prediction error increase of MLP3 is small, and the prediction error increase of MLP1 and MLP2 is large, and at a missing rate of 90%, the prediction error of MLP1 is nearly 6.7% higher than the prediction error of MLP3, and the prediction error of MLP2 is nearly 4.5% higher than the prediction error of MLP3. Therefore, according to the experimental results, it can be seen that the model obtained by the method proposed by the present invention has a lower prediction error for data under different missing rates. Although the prediction error increases with the increase of the missing rate, its increase is small, which can reflect the stability of the model.

[0096] In summary, a method for training a student model proposed in the present invention has the following advantages: by setting a variety of missing rates to perform missing processing on the original samples to simulate the missing data collected in the actual application scenario, the total loss function obtained by weighted summing the representation loss function determined by the representation of the original samples and the representation of the missing samples, the prediction result loss function determined by the prediction results of the original samples and the student prediction results of the missing samples, the contrast loss function determined by the representation of the missing samples, and the absolute error loss function determined by the prediction results of the missing samples and the corresponding labels can not only reduce the difference between the representation of the missing samples and the representation of the original samples, but also the prediction results of the missing samples. The difference between the predicted results and the original samples can be eliminated, and the distance between missing samples with different missing rates belonging to the same original sample can be shortened, and the distance between missing samples that do not belong to the same original sample can be widened; the student model updated according to the total loss function has better generalization ability, and its prediction results are more accurate whether it is for the prediction of complete data or missing data, especially for the prediction of missing data, and it has good prediction accuracy for data with different missing rates. The prediction performance of the model is also more stable, which solves the problem that the existing model cannot predict data with multiple missing rates and has low prediction accuracy, making the model obtained by this method more valuable for practical application.

[0097] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0098] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0099] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0100] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a student model, characterized in that: The method comprises: Obtain a first training set, which includes a plurality of original samples that have not been processed for missing values ​​and a label corresponding to each original sample, wherein the original sample is a historical observation value, and the label is a true value observed after the corresponding historical observation value; Obtaining a teacher model trained using the original samples and labels, the teacher model comprising a feature extractor for extracting teacher representations from the original samples and a regression layer for obtaining teacher prediction results based on the teacher representations; Using multiple missing rates, each original sample in the first training set is processed for missingness, and the corresponding missing samples are obtained. All missing samples and corresponding labels form the second training set. Obtaining a student model, which includes a feature extractor for extracting a student representation from the missing samples and a regression layer for obtaining a student prediction result based on the student representation; The student model is trained using the second training set, the teacher representation, and the teacher prediction result, wherein during the training, the parameters of the student model are updated based on a preset total loss function to obtain a trained student model, and the total loss function includes a weighted sum of the following loss functions: Representation loss function, used to determine the loss between teacher representation and student representation; The prediction result loss function is used to determine the loss between the teacher's prediction result and the student's prediction result; Contrastive loss function, used to determine the contrastive loss between the student representations of different missing samples corresponding to the same original sample and between the student representations of missing samples corresponding to different original samples; The absolute error loss function is used to determine the loss between the student's prediction results and the corresponding labels.

2. The method according to claim 1, characterized in that The total loss function is: in, represents the representation loss function, represents the prediction result loss function, represents the contrast loss function, represents the absolute error loss function, α, β, γ, and λ represent the weights corresponding to the representation loss function, prediction result loss function, contrast loss function, and absolute error loss function, respectively.

3. The method according to claim 2, characterized in that The representation loss function is: Among them, N represents the number of original samples, m represents the number of missing rates, represents the teacher representation of the nth original sample, represents the student representation of the i-th missing sample corresponding to the n-th original sample, and Mean(·) represents the mean of the tensor.

4. The method according to claim 3, characterized in that The prediction result loss function is: Among them, N represents the number of original samples, m represents the number of missing rates, represents the teacher's prediction result of the nth original sample, represents the student prediction result of the i-th missing sample corresponding to the n-th original sample, and Mean(·) represents the mean of the tensor.

5. The method according to claim 2, characterized in that: The contrast loss is determined as follows: Obtain the student representations corresponding to all missing samples, and take the student representations of all missing samples corresponding to each missing rate as a representation sample set; Every two representation sample sets are grouped into sample groups, and anchor samples, positive samples and negative samples are determined from each sample group, where positive samples and anchor samples are student representations corresponding to missing samples from the same original sample but with different missing rates, and negative samples and anchor samples are student representations corresponding to missing samples from different original samples; Calculate the sub-loss of each anchor sample based on the positive and negative samples corresponding to the anchor sample; Determine the sub-contrast loss of each sample group according to the sub-loss of all samples in each sample group; The contrast loss is determined according to the sub-contrastive losses of all sample groups based on a preset contrast loss function.

6. The method according to claim 5, characterized in that The contrast loss function is expressed as: Among them, m represents the number of missing rates, PQ represents the sample group consisting of the Pth characterization sample set and the Wth characterization sample set, and L PQ represents the sub-contrastive loss of sample group PQ.

7. The method according to any one of claims 1 to 6, characterized in that: When the student model is a meteorological forecast model, the historical observation value is a multivariate time series consisting of meteorological data at different locations, and the label is a true meteorological value observed after the corresponding historical observation value; or, When the student model is a traffic flow prediction model, the historical observation value is a multivariate time series consisting of traffic flow data of multiple intersections, and the label is a true value of the traffic flow data observed after the corresponding historical observation value.

8. A method for realizing prediction using a trained student model obtained by the method according to any one of claims 1 to 7, comprising: Get historical observations; The historical observation values ​​are input into the trained student model for prediction to obtain prediction results.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of one of claims 1 to 8 by executing the executable instructions.

Images