A method and apparatus for computing index data

Abstract

The application discloses a kind of methods and devices of calculating index data, it is related to big data technical field.The specific embodiment of the method includes: screening out sample time series data similar to to-be-measured time series data, feature extraction is carried out to the sample time series data, to build data set;Wherein, the index corresponding to the to-be-measured time series data is different from the index corresponding to the sample time series data, and the entry quantity of the to-be-measured time series data is less than the entry quantity of the sample time series data;Using the data set, index calculation model is trained, and the index calculation model after training is obtained;From the to-be-measured time series data, extract to-be-measured feature, input the to-be-measured feature into the index calculation model after training, to output index data.The embodiment can solve the technical problems of large manpower and time consumption and sparse data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a method and device for calculating index data. BACKGROUND

[0002] With the development of big data and intelligent systems, cities tend to be intelligent, and index calculation is an indispensable part of city intelligence, which plays a crucial role in the analysis and judgment of city development, resource scheduling, etc.

[0003] In the process of implementing the present application, the inventors have found that the prior art has at least the following problems:

[0004] 1) Large amount of manpower and time consumption: there are numerous indexes in a city, and the commonly used method is to design a machine learning or deep learning model for each index to fit the data (training phase), thereby completing the calculation of the index (inference phase). For a specific demand scenario, such as social retail sales prediction, sales prediction, etc., the data collection stage usually accesses data from subsystems, the feature extraction stage usually uses a sliding window, the model training usually uses existing open source algorithm libraries such as sklearn, and the model inference stage uses the latest features to calculate the index. The above process requires human experience to model each index, which often requires a huge amount of manpower and time consumption.

[0005] 2) Data sparsity: the indexes in the city are mostly macro indexes, and the time period of historical data is limited. In the case of data sparsity, it is usually difficult to use complex machine learning or deep learning models. SUMMARY

[0006] Therefore, the embodiments of the present application provide a method and device for calculating index data to solve the technical problems of large manpower and time consumption and data sparsity.

[0007] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present application, a method for calculating index data is provided, comprising:

[0008] Filtering out sample time series data similar to the to-be-tested time series data, performing feature extraction on the sample time series data, and thereby constructing a data set; wherein the index corresponding to the to-be-tested time series data is different from the index corresponding to the sample time series data, and the number of entries of the to-be-tested time series data is less than the number of entries of the sample time series data;

[0009] Training the index calculation model using the data set to obtain a trained index calculation model;

[0010] extracting a to-be-tested feature from the to-be-tested time series data, inputting the to-be-tested feature into the trained index calculation model, and outputting index data.

[0011] Optionally, the sample time series data similar to the to-be-tested time series data is screened out, including:

[0012] inputting the to-be-tested time series data and each sample time series data into the trained encoder, and respectively outputting an encoding vector corresponding to the to-be-tested time series data and an encoding vector corresponding to each sample time series data;

[0013] performing clustering on the encoding vectors corresponding to the sample time series data by using a clustering algorithm, to obtain a plurality of clusters and a feature center vector corresponding to each cluster;

[0014] based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, screening out a plurality of sample time series data similar to the to-be-tested time series data.

[0015] Optionally, the number of the clusters is a square root of a total number of the sample time series data.

[0016] Optionally, based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, screening out a plurality of sample time series data similar to the to-be-tested time series data, includes:

[0017] respectively calculating similarities between the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, and screening out N clusters with the largest similarities to the to-be-tested time series data;

[0018] respectively calculating similarities between the encoding vector corresponding to the to-be-tested time series data and the encoding vectors corresponding to the sample time series data in the N clusters, and screening out M sample time series data with the largest similarities to the to-be-tested time series data;

[0019] wherein N is less than M, and N and M are positive integers.

[0020] Optionally, the sample time series data is subjected to feature extraction, to construct a data set, including:

[0021] for each sample time series data, performing feature extraction on each time point in the sample time series data by using a sliding window, to respectively obtain a sample feature and a sample label corresponding to each time point; wherein the sample feature includes the time point and index data before the time point, and the sample label includes index data after the time point;

[0022] based on the sample features and the sample labels corresponding to each time point in the sample time series data, constructing a data set.

[0023] Optionally, the sample feature further comprises a time feature corresponding to the time point.

[0024] Optionally, the data set is used to train the index calculation model to obtain a trained index calculation model, including:

[0025] The data set is divided into a training data set, a verification data set and a test data set;

[0026] The training data set and the verification data set are used to calculate the optimal parameters of each model based on a grid search algorithm and a TPE search algorithm, so as to obtain the index calculation model.

[0027] The test set is used to test the parameter-adjusted index calculation model, so as to filter out an index calculation model with the optimal test result.

[0028] The test set is used to test the parameter-adjusted index calculation model, so as to filter out an index calculation model with the optimal test result.

[0029] Optionally, the data set is used to train the index calculation model to obtain a trained index calculation model, including:

[0030] The sample feature comprises the time point and index data before the time point, and the sample label comprises index data after the time point.

[0031] The sample feature comprises the time point and index data before the time point, and the sample label comprises index data after the time point.

[0032] Optionally, the sample feature further comprises a time feature corresponding to the time point.

[0033] In addition, according to another aspect of the embodiment of the present application, a device for calculating index data is provided, including:

[0034] The filtering module is configured to filter out sample time series data similar to the to-be-tested time series data, extract features of the sample time series data, and construct a data set, wherein the index corresponding to the to-be-tested time series data is different from the index corresponding to the sample time series data, and the number of entries of the to-be-tested time series data is less than the number of entries of the sample time series data.

[0035] a training module configured to train the index calculation model by using the data set, and obtain a trained index calculation model;

[0036] a calculation module configured to extract a to-be-tested feature from the to-be-tested time series data, input the to-be-tested feature into the trained index calculation model, and output index data.

[0037] Optionally, the screening module is further configured to:

[0038] input the to-be-tested time series data and each sample time series data into the trained encoder, and respectively output an encoding vector corresponding to the to-be-tested time series data and an encoding vector corresponding to each sample time series data;

[0039] perform clustering on the encoding vectors corresponding to the sample time series data by using a clustering algorithm, and obtain a plurality of clusters and a feature center vector corresponding to each cluster;

[0040] based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, screen out a plurality of sample time series data similar to the to-be-tested time series data.

[0041] Optionally, the number of the clusters is a square root of a total number of the sample time series data.

[0042] Optionally, the screening module is further configured to:

[0043] respectively calculate similarities between the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, and screen out N clusters with the largest similarities to the to-be-tested time series data;

[0044] respectively calculate similarities between the encoding vector corresponding to the to-be-tested time series data and the encoding vectors corresponding to each sample time series data in the N clusters, and screen out M sample time series data with the largest similarities to the to-be-tested time series data;

[0045] wherein N is less than M, and N and M are positive integers.

[0046] Optionally, the screening module is further configured to:

[0047] for each sample time series data, perform feature extraction on each time point in the sample time series data by using a sliding window, and respectively obtain a sample feature corresponding to each time point and a sample label; wherein the sample feature includes the time point and index data before the time point, and the sample label includes index data after the time point;

[0048] construct a data set based on the sample features and the sample labels corresponding to each time point in each sample time series data.

[0049] Optionally, the training module is further configured to:

[0050] The dataset is divided into a training dataset, a validation dataset, and a test dataset;

[0051] Using the training dataset and the validation dataset, and based on the grid search algorithm and the TPE search algorithm, the optimal parameters of each model are calculated, thereby obtaining the calculation model for each index.

[0052] The parameters of the calculation models for each indicator are adjusted using the time series data to be tested, thereby obtaining the parameter-adjusted calculation models for each indicator.

[0053] The test set is used to test the various index calculation models after parameter adjustment, thereby selecting the index calculation model with the best test results.

[0054] Optionally, the training module is further configured to:

[0055] A sliding window is used to extract features from each time point in the time series data to be tested, and sample features and sample labels corresponding to each time point are obtained respectively; wherein, the sample features include the index data at the time point and before the time point, and the sample labels include the index data after the time point;

[0056] By using the sample features and sample labels corresponding to each time point in the time series data to be tested, the parameters of the calculation models of each indicator are adjusted to obtain the calculation models of each indicator after parameter adjustment.

[0057] According to another aspect of the present invention, an electronic device is also provided, comprising:

[0058] One or more processors;

[0059] Storage device for storing one or more programs.

[0060] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0061] According to another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0062] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0063] An embodiment of the above application has the following advantages or beneficial effects: because the technical means of screening sample time series data similar to the to-be-tested time series data, performing feature extraction on the sample time series data, and constructing a data set are adopted, the technical problems of large consumption of manpower and time and sparse data in the prior art are overcome. The embodiment of the application solves the problem of sparse data by performing feature extraction on sample time series data similar to the to-be-tested time series data and constructing a data set, so that even a complex machine learning or deep learning model can be used, thereby effectively improving the calculation accuracy of the model; and the input of manpower resource cost and time resource cost can also be effectively reduced.

[0064] Further effects of the above non-conventional optional mode will be described below in conjunction with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative effort. Among them:

[0066] Figure 1 is a schematic diagram of the main process of the method for calculating index data according to an embodiment of the application;

[0067] Figure 2 is a schematic diagram of screening sample time series data according to an embodiment of the application;

[0068] Figure 3 is a schematic diagram of extracting features from sample time series data using a sliding window according to an embodiment of the application;

[0069] Figure 4 is a schematic diagram of the main process of the method for calculating index data according to an embodiment of the application;

[0070] Figure 5 is a schematic diagram of the main modules of the device for calculating index data according to an embodiment of the application;

[0071] Figure 6 is an exemplary system architecture diagram to which the embodiment of the application can be applied;

[0072] Figure 7 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing the embodiment of the application. DETAILED DESCRIPTION

[0073] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various specific details to facilitate understanding, and are to be considered in conjunction with the description. Thus, one of ordinary skill in the art will appreciate that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the application. Similarly, for the sake of clarity and conciseness, the description below will not describe all possible combinations of the features described herein.

[0074] Figure 1 is a schematic diagram of the main flow of the method for calculating index data according to an embodiment of the present application. As shown in Figure 1 the method for calculating index data can include:

[0075] Step 101: Screening sample time series data similar to the time series data to be measured, and performing feature extraction on the sample time series data to construct a data set.

[0076] To solve the problem of data sparsity, an embodiment of the present application screens sample time series data similar to the time series data to be measured, and performs feature extraction on the sample time series data to construct a data set for training the model; wherein the index corresponding to the time series data to be measured is different from the index corresponding to the sample time series data, and the number of entries of the time series data to be measured is less than the number of entries of the sample time series data. Since the number of entries of the time series data to be measured is small (less than a preset entry number threshold, such as less than 500 entries, or less than 100 entries, etc.; wherein the entry number threshold can be preset according to business needs), it is necessary to screen sample time series data similar to the time series data to be measured, and use these sample time series data as training samples.

[0077] Step 101 can be divided into two steps, the first step is to screen M sample time series data similar to the time series data to be measured from existing historical data, and the second step is to perform feature extraction on the M sample time series data screened.

[0078] Specifically, data needs to be collected first, and the data is usually collected from various business subsystems. The format of each piece of data is as follows: each piece of data contains a time field and a plurality of dimension fields (the number of dimension fields is greater than or equal to 1, and the dimension represents the index to be calculated, such as sales volume, sales amount, etc.).

[0079] Time Dimension 1 Dimension 2 Dimension 3 Dimension 4 Dimension 5 Dimension 6 2021-10-01 *** *** *** *** *** *** 2021-10-08 *** *** *** *** *** *** 2021-11-01 *** *** *** *** *** ***

[0080] Optionally, the sample time series data similar to the to-be-tested time series data is screened out, including: inputting the to-be-tested time series data and each sample time series data into a trained encoder, and respectively outputting an encoding vector corresponding to the to-be-tested time series data and an encoding vector corresponding to each sample time series data; using a clustering algorithm to cluster the encoding vectors corresponding to the sample time series data, to obtain a plurality of clusters and a feature center vector corresponding to each cluster; and based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, screening out a plurality of sample time series data similar to the to-be-tested time series data.

[0081] As shown in the formula (1), the to-be-tested time series data and each sample time series data are input into a trained encoder (Encoder), and n encoding vectors are obtained assuming that there are n historical time series data. Then a clustering algorithm (such as a K-Means clustering algorithm) is used to cluster the encoding vectors corresponding to each sample time series data, to obtain a plurality of clusters and a feature center vector corresponding to each cluster. Finally, based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, a plurality of sample time series data similar to the to-be-tested time series data are screened out. Figure 2

[0082] It should be noted that the encoder needs to be trained in advance. An encoder (Encoder) and a decoder (Decoder) are trained through a self-encoding model. The Encoder receives time series data x as input, generates an encoding vector f = f(x) ∈ R d , and d is the dimension of the encoding vector, which can be 128. The Decoder is responsible for restoring the original input data from the encoding vector f. After obtaining the encoder, the encoder can be used to encode each historical sample time series data.

[0083] Optionally, the number of clusters is the square root of the total number of the sample time series data. Assuming that n sample time series data are clustered, a total of clusters and their corresponding feature center vectors can be formed. The initial point of clustering can be set in advance, such as setting the initial point to according to the total number of sample time series data, so that the calculation speed of similarity in the subsequent steps can be improved.

[0084] ​Optionally, screening a plurality of sample time series data similar to the to-be-tested time series data based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the plurality of clusters comprises: calculating the similarity of the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the plurality of clusters respectively, and screening N clusters with the largest similarity to the to-be-tested time series data; calculating the similarity of the encoding vector corresponding to the to-be-tested time series data and the encoding vector corresponding to each sample time series data in the N clusters respectively, and screening M sample time series data with the largest similarity to the to-be-tested time series data; wherein N is less than M, and N and M are positive integers.

[0085] As shown in FIG. 1, for a to-be-tested time series data, an encoding vector g = g(x) is first generated by an encoder, then the similarity between the g vector and the feature center vectors corresponding to each cluster is calculated respectively, and the two feature center vectors U and V with the largest similarity are screened from the feature center vectors, then the similarity between the g vector and the two feature center vectors U and V is calculated respectively, and the 10 vectors with the largest similarity are selected, that is, Figure 2

[0086] v * = argmin sim(g, v)

[0087] v∈U|V

[0088] Therefore, the 10 sample time series data most similar to the to-be-tested time series data can be obtained through the above process. It should be noted that the values of N and M can be pre-set, and the values of N and M are exemplarily shown in the embodiments of the present application, but the values shown in the embodiments are not limited.

[0089] The prior art generally needs to calculate the similarity between the to-be-tested time series data and each sample time series data, and the time complexity of the calculation is O(n), while the time complexity of the embodiments of the present application is Thus, the time complexity of comparing the similarity with each sample time series data one by one can be reduced by one rough sorting and two fine sortings.

[0090] Optionally, feature extraction is performed on the sample time series data to construct a data set, comprising: for each sample time series data, performing feature extraction on each time point in the sample time series data by using a sliding window to obtain sample features and sample labels corresponding to each time point respectively; wherein the sample features include the time point and index data before the time point, and the sample labels include index data after the time point; and constructing a data set based on the sample features and sample labels corresponding to each time point in the plurality of sample time series data.

[0091] As shown in FIG. 2, for each sample time series data, a sliding window is used to perform feature extraction on each time point in the sample time series data to obtain sample features and sample labels corresponding to each time point. Figure 3 ​As shown, the sample time series data is extracted by using a sliding window, and the sample time series data to be extracted is Each time point contains d f dimensional features, the size of the sliding window is s, and the step length to be predicted is l, then the sample features extracted at time point i are (x i-s+1 ,…,x i-1 ,x i ), and the sample label is (x i+1 ,x i+2 ,…,x i+l ). It should be noted that since the step length to be predicted is l, in order to ensure the accuracy of the calculation, the length of the sample label is preferably also l.

[0092] Optionally, the sample features further include time features corresponding to the time points. In an embodiment of the present application, in order to improve the calculation accuracy, the time features corresponding to each time point in the sample time series data also need to be extracted. Specifically, first, the dimensional information in the time is extracted, including the month, day, week, day number in the year, week number in the year, quarter, and whether the date is a weekday or a holiday corresponding to the current time point, etc. If the granularity of the time information is smaller, such as minute or hour granularity (traffic flow prediction scenario, passenger flow prediction scenario), then the hour information, minute information, etc. can also be extracted, and the embodiments of the present application do not limit this. Then, the features extracted by the sliding window and the time features are spliced together to form a complete sample feature, and the sample feature and the corresponding sample label constitute a sample data.

[0093] For each sample time series data, a plurality of sample data can be extracted by using a sliding window, so each sample data corresponding to each sample time series data collectively constitutes a data set.

[0094] Step 102, using the data set to train the index calculation model to obtain a trained index calculation model.

[0095] After obtaining the data set, the sample data in the data set is used to train the index calculation model, the index calculation model is fitted to the sample data, and finally a trained index calculation model is obtained.

[0096] Optionally, step 102 can include: dividing the data set into a training data set, a validation data set and a test data set; using the training data set and the validation data set and based on a grid search algorithm and a TPE search algorithm, calculating optimal parameters of each model, thereby obtaining the index calculation model; using the to-be-tested time series data to perform parameter adjustment on the each index calculation model, thereby obtaining the each index calculation model after parameter adjustment; using the test set to test the each index calculation model after parameter adjustment, thereby screening out an index calculation model with the optimal test result. Specifically, the data set can be divided into a training data set D train , a validation data set D val and a test data set D test ; wherein the training data set is used to learn parameters Θ of the model m; the validation data set is used to select the model m (that is, to select a model with the best performance from multiple models, for a prediction task, the model can be selected using MSE, and optimal parameters Θ of the model * ; and the test data set is used to test the final effect of the model.

[0097] Generally, the division mode of the data set is equal proportion division according to time sequence, for example, training data: validation data: test data = 8:1:1.

[0098] In order to improve the accuracy of the training model, the embodiment of the application uses a hyperparameter optimization method to search for the optimal result of the training model. In the process of hyperparameter optimization, the model space (which models can be selected) and the parameter space corresponding to the model (which space can each model hyperparameter be selected from) are important. Optionally, the model space and the corresponding parameter space contained in the embodiment of the application are as follows:

[0099] ARIMA model: the ARIMA model is a difference moving autoregressive model, a stationary sequence is obtained through difference (Δ x = x i+1 -x i ), and then an autoregressive model is used to fit the linear relationship between the current data and the historical sliding window data. For the ARIMA model, the parameter space contained includes the number of differences (d), the length of autoregression (p), etc.

[0100] Ridge model: the Ridge model uses a linear model with a regularization term to fit the training data, wherein the model assumption is: and the loss function is wherein d is the dimension of the data feature, the feature of the sample is (x1, x2, …, x d ), y i and where y is the corresponding sample label and yhat is the corresponding predicted value, and a is the regularization coefficient. For Ridge model, the parameter space includes the regularization coefficient a.

[0101] RandomForest model: RandomForest is a random forest model that integrates multiple decision trees to complete prediction. The parameter space of the model includes the number of trees, the proportion of split point sampling, the minimum sample number of each leaf node, etc.

[0102] Xgboost model: xgboost is a boosting tree model, each tree fits the residual of the existing tree model and label data, i.e. The parameter space of the model includes the number of trees, learning rate, regularization term, etc.

[0103] For the search of these parameters, the embodiment of the present application uses a grid search algorithm and a TPE (Tree-structured Parzen Estimator) search algorithm. Among them, the grid search algorithm enumerates each possible value of each parameter, and completes the combination of parameters through permutation and combination. For each parameter combination, the training data set D train The parameters of the model are trained, and the validation data set D val is used to complete the validation of the model effect. Assuming that there are two parameters a and b, and each parameter has possible values (a1, a2, a3) and (b1, b2), then the parameter groups that need to be enumerated in the grid search algorithm include (a1, b1), (a1, b2), (a2, b1), (a2, b2), (a3, b1), (a3, b2).

[0104] The TPE search algorithm is to construct a sample pair (θ i , val val ) for each parameter group θ i and its evaluation index val i on the validation set D i , and then fit the data set y = f (θ; θ i , val i ) composed of parameter combination evaluation indexes through Gaussian process, and infer the parameter θ * = argmax f (θ; θ i , val i ) corresponding to the maximum evaluation index. Repeat the above process until the optimal parameter θ * is obtained, or the stopping condition of the algorithm is reached.

[0105] The embodiment of the present application adopts the combination of the grid search algorithm and the TPE search algorithm, and the main reasons for consideration have the following two aspects: 1. In the case of less parameter groups to be searched, the grid search algorithm is more efficient; 2. The TPE algorithm can make up for the deficiency of the grid search algorithm, explore the possible optimal solution in more space, and mainly solve the problem of insufficient search space caused by equal division of the grid search algorithm.

[0106] Optionally, the parameter adjustment of the various index calculation models is performed by using the to-be-tested time series data, so as to obtain the various index calculation models after parameter adjustment, including: feature extraction of each time in the to-be-tested time series data by using a sliding window, to obtain sample features and sample labels corresponding to each time, respectively; wherein the sample features include the time and index data before the time, and the sample labels include index data after the time; the parameter adjustment of the various index calculation models is performed by using the sample features and sample labels corresponding to each time in the to-be-tested time series data, so as to obtain the various index calculation models after parameter adjustment.

[0107] As shown in Figure 3 , similarly to the feature extraction of the sample time series data, the features of each time in the to-be-tested time series data are extracted by using a sliding window, the size of the sliding window is s, the step length to be predicted is l, the sample features extracted at time i are (x i-s+1 ,…,x i-1 ,x i ), and the sample labels are (x i+1 ,x i+2 ,…,x i+l ). Then the parameter fine-tuning of the various index calculation models is performed by using the sample features and sample labels (that is, the training set) corresponding to each time in the to-be-tested time series data. Assuming that the parameters of the index calculation model with the optimal test result are , the parameters of the fine-tuned model are wherein θ * is the optimal parameter of the fine-tuned model, η is the learning rate of the model parameter in the fine-tuning stage, L is the loss function of MSE, used to measure the error of the model on the training set, and (x, y) is the sample data in the training set. Optionally, in the specific implementation process, the mini batch gradient descent (mini batch SGD) can be used.

[0108] Since the index calculation model used in the embodiment of the present application is trained only by using single-dimensional data, when the model is fine-tuned, the model fine-tuning is not performed on multi-dimensional data.

[0109] Optionally, the sample feature further comprises a time feature corresponding to the time point. Similar to extracting the feature of the sample time series data, the time feature corresponding to each time point in the sample time series data also needs to be extracted. Specifically, first, the dimension information in time is extracted, including the month corresponding to the current time point, the day, the week, the day number of the date in the year, the week number of the date in the year, the quarter of the date, whether the date is a weekday or a holiday, etc. If the granularity of the time information is smaller, such as minute or hour granularity (traffic flow prediction scene, passenger flow prediction scene), then the hour information, minute information, etc. can also be extracted, and the embodiments of the present application do not limit this. Then, the features extracted by the sliding window and the time features are spliced together to form a complete sample feature, and the sample feature and the corresponding sample label constitute a sample data.

[0110] In order to prevent overfitting of the verification set, finally, the test data set D test The effects of different models are tested, and the model with the optimal result is selected, wherein the evaluation index used is y is the sample label, is the prediction result of the model. After model verification, the optimal index calculation model

[0111] Step 103, extracting the to-be-tested feature from the to-be-tested time series data, inputting the to-be-tested feature into the trained index calculation model, and outputting the index data.

[0112] For the to-be-tested index, the calculation of the index data usually needs to extract features from the last period of data, and the extracted features are denoted as x p (i.e. to-be-tested feature), and then using the optimal index calculation model to calculate the index data, and obtaining is the output result of the index calculation model.

[0113] According to the various embodiments described above, it can be seen that the embodiments of the present application solve the technical problems of large consumption of manpower and time and sparse data in the prior art by screening sample time series data similar to the to-be-tested time series data, extracting features from the sample time series data, and constructing a data set. The embodiments of the present application solve the problem of sparse data by extracting features from the sample time series data similar to the to-be-tested time series data and constructing a data set, so that even complex machine learning or deep learning models can be used, thereby effectively improving the calculation accuracy of the model. Moreover, the investment of human resource cost and time resource cost can also be effectively reduced.

[0114] Figure 4is a schematic diagram of the main flow of a method for calculating index data according to one embodiment of the present application. As another embodiment of the present application, as shown in Figure 4 the method for calculating index data can include:

[0115] Step 401, collect historical sample time series data.

[0116] Specifically, for each index, the sample time series data corresponding to the index is collected.

[0117] Step 402, select sample time series data similar to the to-be-tested time series data from the sample time series data.

[0118] The to-be-tested time series data corresponds to an index different from the index corresponding to the sample time series data, and the number of entries of the to-be-tested time series data is less than the number of entries of the sample time series data.

[0119] Optionally, step 402 can include: inputting the to-be-tested time series data and the sample time series data into a trained encoder to respectively output an encoding vector corresponding to the to-be-tested time series data and an encoding vector corresponding to the sample time series data; clustering the encoding vectors corresponding to the sample time series data using a clustering algorithm to obtain a plurality of clusters and a feature center vector corresponding to each cluster (first rough sorting); and selecting a plurality of sample time series data similar to the to-be-tested time series data based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters (second fine sorting). The embodiment of the present application can reduce the time complexity of comparing the similarity of each sample time series data one by one through first rough sorting and second fine sorting.

[0120] Step 403, performing feature extraction on the sample time series data to construct a data set, and performing feature extraction on the to-be-tested time series data to obtain fine-tuning sample data.

[0121] In the embodiment of the present application, the sample time series data and the to-be-tested time series data are extracted using a sliding window to construct the data set and the fine-tuning sample data. The data set is used to train the models, and the fine-tuning sample data is used to fine-tune the parameters of the models.

[0122] Step 404, divide the data set into a training data set, a validation data set, and a test data set.

[0123] Step 405, calculate the optimal parameters of each model using the training data set and the validation data set based on a grid search algorithm and a TPE search algorithm to obtain the index calculation model.

[0124] The grid search algorithm enumerates each possible value of each parameter, and the parameter combination is completed by permutation and combination. The training data set D train The parameters of the model are trained, and the validation data set D val The validation of the model effect is completed

[0125] Step 406, using the fine-tuning sample data corresponding to the to-be-tested time series data, fine-tuning the parameters of each index calculation model, thereby obtaining the fine-tuned index calculation model.

[0126] Step 407, using the test set to test the fine-tuned index calculation model, thereby screening out the index calculation model with the optimal test result.

[0127] In order to prevent overfitting of the validation set, finally use the test data set D test Test the effects of different models, and select the model with the optimal result,

[0128] Step 408, extracting the to-be-tested feature from the to-be-tested time series data, inputting the to-be-tested feature into the trained index calculation model, and outputting the index data.

[0129] Extract the to-be-tested feature from the last period of time of the to-be-tested time series data, denoted as x p Then use the optimal index calculation model Calculate the index data to obtain That is, the output result of the index calculation model.

[0130] In addition, the specific implementation content of the method for calculating the index data in one of the reference embodiments of the present application has been described in detail in the above method for calculating the index data, and therefore the repeated content will not be described here.

[0131] Figure 5 is a schematic diagram of the main modules of the device for calculating the index data according to the embodiments of the present application. As Figure 5As shown, the device 500 for calculating index data comprises a screening module 501, a training module 502 and a calculation module 503; the screening module 501 is configured to screen sample time series data similar to to-be-tested time series data, perform feature extraction on the sample time series data, and thus construct a data set; the index corresponding to the to-be-tested time series data is different from the index corresponding to the sample time series data, and the number of entries of the to-be-tested time series data is less than the number of entries of the sample time series data; the training module 502 is configured to train an index calculation model by using the data set, and obtain a trained index calculation model; the calculation module 503 is configured to extract to-be-tested features from the to-be-tested time series data, input the to-be-tested features into the trained index calculation model, and thus output index data.

[0132] Optionally, the screening module 501 is further configured to:

[0133] input the to-be-tested time series data and each sample time series data into a trained encoder, and respectively output an encoding vector corresponding to the to-be-tested time series data and an encoding vector corresponding to each sample time series data;

[0134] perform clustering on the encoding vectors corresponding to the sample time series data by using a clustering algorithm, and obtain a plurality of clusters and a feature center vector corresponding to each cluster;

[0135] based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, screen a plurality of sample time series data similar to the to-be-tested time series data.

[0136] Optionally, the number of the clusters is a square root of the total number of the sample time series data.

[0137] Optionally, the screening module 501 is further configured to:

[0138] respectively calculate the similarity of the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to the clusters, and screen N clusters with the largest similarity to the to-be-tested time series data;

[0139] respectively calculate the similarity of the encoding vector corresponding to the to-be-tested time series data and the encoding vectors corresponding to each sample time series data in the N clusters, and screen M sample time series data with the largest similarity to the to-be-tested time series data;

[0140] wherein N is less than M, and N and M are positive integers.

[0141] Optionally, the screening module 501 is further configured to:

[0142] For each sample time series data, a sliding window is used to extract features of each time point in the sample time series data, and sample features and sample labels corresponding to each time point are obtained respectively; wherein the sample features include the time point and index data before the time point, and the sample labels include index data after the time point;

[0143] Based on the sample features and sample labels corresponding to each time point in the respective sample time series data, a data set is constructed.

[0144] Optionally, the training module 502 is further configured to:

[0145] divide the data set into a training data set, a validation data set and a test data set;

[0146] using the training data set and the validation data set and based on a grid search algorithm and a TPE search algorithm, calculate optimal parameters of each model, thereby obtaining the index calculation models;

[0147] using the to-be-tested time series data, adjust parameters of the index calculation models, thereby obtaining the index calculation models after parameter adjustment;

[0148] using the test set, test the index calculation models after parameter adjustment, thereby screening out an index calculation model with the optimal test result.

[0149] Optionally, the training module 502 is further configured to:

[0150] using a sliding window, extract features of each time point in the to-be-tested time series data, and obtain sample features and sample labels corresponding to each time point respectively; wherein the sample features include the time point and index data before the time point, and the sample labels include index data after the time point;

[0151] using the sample features and sample labels corresponding to each time point in the to-be-tested time series data, adjust parameters of the index calculation models, thereby obtaining the index calculation models after parameter adjustment.

[0152] It should be noted that the specific implementation of the device for calculating index data described in the present application has been described in detail in the above method for calculating index data, and therefore the repeated content will not be described here.

[0153] Figure 6 An exemplary system architecture 600 is shown, which can apply the method for calculating index data or the device for calculating index data according to the embodiments of the present application.

[0154] As Figure 6As shown, the system architecture 600 can include terminal devices 601, 602, 603, a network 604 and a server 605. The network 604 is a medium for providing communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0155] The users can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0156] The terminal devices 601, 602, 603 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0157] The server 605 can be a server providing various services, such as a background management server supporting shopping websites browsed by users using the terminal devices 601, 602, 603 (only as an example). The background management server can analyze and process received item information query requests and other data, and feed back the processing results to the terminal devices.

[0158] It should be noted that the method for calculating index data provided by the embodiments of the present application is generally executed by the server 605, and accordingly, the device for calculating index data is generally provided in the server 605. The method for calculating index data provided by the embodiments of the present application can also be executed by the terminal devices 601, 602, 603, and accordingly, the device for calculating index data can be provided in the terminal devices 601, 602, 603.

[0159] It should be understood that, Figure 6 The number of terminal devices, networks and servers in the system architecture 600 is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks and servers.

[0160] Reference will be made to Figure 7 which shows a structural schematic diagram of a computer system 700 of a terminal device suitable for implementing the embodiments of the present application. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0161] As Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage section 708. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0162] Connected to the I / O interface 705 are an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable recording medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as necessary, so that a computer program read therefrom is installed into the storage section 708 as necessary.

[0163] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable recording medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-described functions defined in the system of the present disclosure are executed.

[0164] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or instrument, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or instrument. In the present application, the computer-readable signal medium can include a data signal propagating in a baseband or as a carrier wave part of a carrier wave, in which computer-readable program code is carried. Such a propagating data signal can take various forms, including but not limited to electromagnetic signals, optical signals or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or instrument. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0165] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer programs according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0166] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The modules described can also be arranged in a processor, for example, a processor can be described as including a screening module, a training module and a calculation module, wherein the names of the modules do not constitute a limitation on the modules themselves in some cases.

[0167] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: screening sample timing data similar to to-be-tested timing data, performing feature extraction on the sample timing data to construct a data set, wherein the to-be-tested timing data corresponds to an index different from an index corresponding to the sample timing data, and the number of entries of the to-be-tested timing data is less than the number of entries of the sample timing data; training an index calculation model using the data set to obtain a trained index calculation model; extracting to-be-tested features from the to-be-tested timing data, inputting the to-be-tested features into the trained index calculation model, and outputting index data.

[0168] As another aspect, the embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the method described in any of the above embodiments.

[0169] According to the technical scheme of the embodiments of the present application, because the sample timing data similar to the to-be-tested timing data is screened, the feature extraction is performed on the sample timing data to construct the data set, the technical problems of large consumption of manpower and time and sparse data in the prior art are overcome. The embodiments of the present application solve the problem of sparse data by performing feature extraction on the sample timing data similar to the to-be-tested timing data to construct the data set, even complex machine learning or deep learning models can be used, so that the calculation accuracy of the model can be effectively improved; moreover, the input of manpower resource cost and time resource cost can be effectively reduced.

[0170] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of calculating index data, characterized by, The method comprises the following steps: Screening sample time series data similar to the to-be-tested time series data, performing feature extraction on the sample time series data, and thereby constructing a data set; wherein the index corresponding to the to-be-tested time series data is different from the index corresponding to the sample time series data, and the number of entries of the to-be-tested time series data is less than the number of entries of the sample time series data; Training the index calculation model using the data set to obtain a trained index calculation model; Extracting to-be-tested features from the to-be-tested time series data, inputting the to-be-tested features into the trained index calculation model, and thereby outputting index data; Screening sample time series data similar to the to-be-tested time series data, comprising: Inputting the to-be-tested time series data and each sample time series data into a trained encoder to respectively output an encoding vector corresponding to the to-be-tested time series data and an encoding vector corresponding to each sample time series data; Using a clustering algorithm to cluster the encoding vectors corresponding to each sample time series data to obtain a plurality of clusters and a feature center vector corresponding to each cluster; Based on the encoding vector corresponding to the to-be-tested time series data and the feature center vector corresponding to each cluster, screening a plurality of sample time series data similar to the to-be-tested time series data.

2. The method of claim 1, wherein, The number of clusters is the square root of the total number of each sample time series data.

3. The method of claim 1, wherein, Based on the encoding vector corresponding to the to-be-tested time series data and the feature center vector corresponding to each cluster, screening a plurality of sample time series data similar to the to-be-tested time series data, comprising: Respectively calculating the similarity of the encoding vector corresponding to the to-be-tested time series data and the feature center vector corresponding to each cluster, and screening N clusters with the greatest similarity to the to-be-tested time series data; Respectively calculating the similarity of the encoding vector corresponding to the to-be-tested time series data and the encoding vector corresponding to each sample time series data in the N clusters, and screening M sample time series data with the greatest similarity to the to-be-tested time series data; Wherein N is less than M, and N and M are positive integers.

4. The method of claim 1, wherein, Performing feature extraction on the sample time series data to thereby construct a data set, comprising: For each sample time series data, performing feature extraction on each time point in the sample time series data using a sliding window to respectively obtain a sample feature and a sample label corresponding to each time point; wherein the sample feature comprises the time point and index data before the time point, and the sample label comprises index data after the time point; Based on the sample feature and the sample label corresponding to each time point in each sample time series data, constructing a data set.

5. The method of claim 4, wherein, The sample feature further comprises a time feature corresponding to the time point.

6. The method of claim 1, wherein, Training the index calculation model using the data set to obtain a trained index calculation model, comprising: Dividing the data set into a training data set, a validation data set, and a test data set; Using the training data set and the validation data set and based on a grid search algorithm and a TPE search algorithm, calculating the optimal parameters of each model to thereby obtain the index calculation models; Using the to-be-tested time series data to adjust the parameters of the index calculation models to thereby obtain the index calculation models with adjusted parameters; The parameter-adjusted index calculation models are tested by using the test set, so as to screen out an index calculation model with optimal test result.

7. The method of claim 6, wherein, The parameter-adjusted index calculation models are obtained by performing parameter adjustment on the index calculation models by using the to-be-tested time series data, including: Features of each time point in the to-be-tested time series data are extracted by using a sliding window, and sample features and sample labels corresponding to each time point are obtained respectively; the sample features include the time point and index data before the time point, and the sample labels include index data after the time point; The parameter-adjusted index calculation models are obtained by performing parameter adjustment on the index calculation models by using the sample features and sample labels corresponding to each time point in the to-be-tested time series data.

8. An apparatus for computing index data, characterized by Including: The screening module is configured to screen sample time series data similar to to-be-tested time series data, extract features of the sample time series data, and construct a data set; the index corresponding to the to-be-tested time series data is different from the index corresponding to the sample time series data, and the number of entries of the to-be-tested time series data is less than the number of entries of the sample time series data; The training module is configured to train an index calculation model by using the data set, and obtain a trained index calculation model; The calculation module is configured to extract to-be-tested features from the to-be-tested time series data, input the to-be-tested features into the trained index calculation model, and output index data; The screening module is further configured to: input the to-be-tested time series data and each sample time series data into a trained encoder, and output an encoding vector corresponding to the to-be-tested time series data and encoding vectors corresponding to the sample time series data respectively; perform clustering on the encoding vectors corresponding to the sample time series data by using a clustering algorithm, and obtain a plurality of clusters and feature center vectors corresponding to each cluster; based on the encoding vector corresponding to the to-be-tested time series data and the feature center vectors corresponding to each cluster, screen out a plurality of sample time series data similar to the to-be-tested time series data.

9. An electronic device, comprising: Including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

10. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-7.

Images