Index recommendation method and device for time sequence prediction task and computing equipment cluster

Abstract

The invention discloses an index recommendation method and device for a time sequence prediction task and a computing device cluster, and belongs to the technical field of computers. The method comprises the steps that a reference time sequence of a time sequence prediction task and a plurality of labeled prediction time sequences are obtained, a to-be-recommended index is determined according to the obtained time sequences, the labeled prediction time sequences comprise labeling information, and the labeled prediction time sequences comprise the labeling information; the annotation information is used for indicating the priority of the time sequence prediction method corresponding to each annotated prediction time sequence, so that the annotation information can accurately reflect the prediction effect of the time sequence prediction method corresponding to each annotated prediction time sequence; the index determined by using the annotation information can accurately evaluate the prediction effect when the time sequence prediction task is executed by adopting different time sequence prediction methods, and the expected conformity of the evaluation result for the time sequence prediction task is improved.

Description

[0001] This application claims priority to Chinese patent application No. 202311344330.8, filed on October 17, 2023, with invention name “A method, device and computing device cluster for indicator recommendation”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an indicator recommendation method, device and computing device cluster for a time series prediction task. Background Art

[0003] Time series forecasting (abbreviated as time series forecasting) is a method of predicting future time series based on the historical time series of events. The time series is a series of values ​​of the same statistical dimension arranged in the order of their occurrence. This method is widely used in a variety of forecasting scenarios, such as material demand forecasting, cost forecasting, temperature forecasting, power load forecasting, etc. At present, the evaluation system evaluates the quality of time series forecasting tasks, which can guide the optimization of time series forecasting models and improve the reliability of time series forecasting results.

[0004] In the related art, the evaluation system provides users with multiple indicators, such as mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), mean absolute percentage error (MAPE), etc., and evaluates the time series prediction method used to perform the time series prediction task based on the indicators selected by the user, and presents the evaluation results to the user.

[0005] However, since users often have limited understanding of the applicability of indicators, they are prone to selecting inappropriate indicators, that is, the indicators are not reference-based, resulting in a situation where the evaluation results based on the indicators are very good but the actual quality is very poor, resulting in low accuracy of the evaluation results, which affects the judgment of the quality of the time series prediction task. Summary of the invention

[0006] The embodiments of the present application provide an indicator recommendation method, apparatus and computing device cluster for a timing prediction task, which can automatically recommend indicators suitable for the timing prediction task, and the indicators can accurately evaluate the prediction effect when the timing prediction task is performed using different timing prediction methods, thereby improving the expected conformity of the evaluation results of the timing prediction task.

[0007] On the first aspect, the present application provides an indicator recommendation method for time series prediction tasks, which is applied to the scenario of evaluating the quality of time series prediction tasks. Since the evaluation process of time series prediction tasks often involves multiple indicators, different indicators are applicable to different types of time series. For example, mean absolute square error (MASE) is applicable to time series with periodic or seasonal characteristics; MAPE is applicable to time series with significant differences in values ​​between different time points and significant influence of outliers; RMSE is more sensitive to outliers, that is, it is applicable to time series with outliers, etc. Based on this, the present application provides an indicator recommendation method for time series prediction tasks, which can automatically recommend indicators applicable to the time series prediction task, or the most suitable indicators for the time series prediction task, based on the relevant time series of the time series prediction task, thereby improving the expected conformity of the evaluation results of the time series prediction task, and the expected conformity means that the evaluation results of the time series prediction task are more in line with the user's expectations.

[0008] Illustratively, a reference time series of a time series prediction task is obtained, and the time series prediction task indicates predicting a time series of events; wherein the time series prediction task is, for example, material demand prediction, cost prediction, temperature prediction, power load prediction, etc., and the present application does not limit the type of time series prediction task.

[0009] Acquire multiple annotated prediction time series of a time series prediction task, where the reference time series and the annotated prediction time series have an overlapping time period, any two annotated prediction time series correspond to different time series prediction methods, and the multiple annotated prediction time series include annotation information, and the annotation information indicates the priority of the time series prediction method corresponding to each annotated prediction time series;

[0010] Based on a reference time series and multiple annotated prediction time series, the index to be recommended is determined. The index is used to evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks.

[0011] Among them, multiple annotated prediction time series include annotation information to indicate the priority of the time series prediction method corresponding to each annotated prediction time series. Therefore, the annotation information can accurately reflect the prediction effect of the time series prediction method corresponding to each annotated prediction time series. Furthermore, the indicators determined by using this annotation information can accurately evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks, thereby improving the expected conformity of the evaluation results for time series prediction tasks.

[0012] In some embodiments, obtaining a reference time series for a time series prediction task includes: obtaining a reference time series uploaded by a user.

[0013] In some embodiments, obtaining a plurality of annotated predicted time series of a time series prediction task includes any of the following:

[0014] Get the predicted time series of multiple annotations uploaded by the user;

[0015] The real time series of the time series prediction task uploaded by the user is obtained, and based on the real time series, a plurality of different time series prediction methods are used to perform the time series prediction task to obtain a plurality of predicted time series, and based on the plurality of predicted time series, annotation information is obtained to obtain a plurality of annotated prediction time series.

[0016] In some embodiments, determining the indicator to be recommended based on the reference time series and the multiple annotated predicted time series includes: determining the overlapping time period of the reference time series and each annotated predicted time series; and determining the indicator to be recommended based on the time series corresponding to the overlapping time period. Further determining the indicator to be recommended by determining the overlapping time period can improve the accuracy of the determined indicator.

[0017] In some embodiments, determining the indicator to be recommended based on the reference time series and the plurality of annotated predicted time series includes:

[0018] Based on the reference time series, the multiple annotated prediction time series and the multiple indicators of the time series prediction task, determining the first recommended reference information of each indicator, wherein the first recommended reference information of the indicator indicates the consistency between the evaluation result obtained by performing priority evaluation on the time series prediction methods corresponding to the multiple annotated prediction time series based on the indicator and the annotated information;

[0019] Based on the first recommendation reference information of each indicator, the indicator to be recommended is determined.

[0020] In some embodiments, based on the reference time series, the plurality of annotated predicted time series, and the plurality of indicators of the time series prediction task, determining the first recommended reference information for each indicator includes:

[0021] Based on the reference time series, the multiple annotated prediction time series, and the first indicator, obtaining an evaluation result corresponding to the first indicator, where the first indicator refers to any one of the multiple indicators;

[0022] Based on the evaluation result and the annotation information corresponding to the first indicator, obtaining a recommended score value of the first indicator, the recommended score value of the first indicator indicating consistency between the evaluation result and the annotation information corresponding to the first indicator;

[0023] Based on the recommendation score value of the first indicator, first recommendation reference information of the first indicator is determined.

[0024] Through the above method, since the annotation information can accurately reflect the accuracy of the predicted time series, the first recommended reference information of the indicator is obtained based on the recommended score value between the evaluation result corresponding to the indicator and the annotation information, so that the first recommended reference information can reflect whether the prediction effect of the time series prediction task can be accurately measured through this indicator according to the consistency between the evaluation result corresponding to the indicator and the annotation information, thereby providing technical support for the subsequent determination of the indicator to be recommended.

[0025] In some embodiments, based on the first recommendation reference information of each indicator, determining the indicator to be recommended includes at least one of the following:

[0026] Based on the first recommendation reference information of each indicator, determining an indicator to be recommended from the plurality of indicators;

[0027] Based on the first recommended reference information of each indicator, multiple candidate indicators are determined from multiple indicators, and the evaluation results corresponding to each candidate indicator are fused. Based on the fused evaluation results, a fused indicator to be recommended is generated. The fused indicator is used to comprehensively evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks from multiple evaluation dimensions.

[0028] That is, the indicator to be recommended can be selected from multiple indicators, or it can be a fusion indicator obtained by fusing multiple indicators. Among them, by selecting the indicator to be recommended from multiple indicators, it is possible to conveniently and quickly select the indicator suitable for the time series prediction task from multiple indicators based on the recommended reference information of each indicator, and use it as the indicator to be recommended, which can improve the processing efficiency on the basis of improving the expected conformity of the evaluation results of the time series prediction task. By generating a fusion indicator, the prediction effect of the time series prediction task can be comprehensively measured from multiple evaluation dimensions, thereby effectively improving the expected conformity of the evaluation results of the time series prediction task.

[0029] In some embodiments, based on the first recommendation reference information of each indicator, multiple candidate indicators are determined from multiple indicators, the evaluation results of each candidate indicator are fused, and based on the fused evaluation results, a fused indicator to be recommended is generated, including:

[0030] Based on the first recommended reference information of each indicator, determining a plurality of candidate indicators from the plurality of indicators;

[0031] Based on the evaluation result corresponding to each candidate indicator and the fusion weight corresponding to each candidate indicator, the evaluation result corresponding to each candidate indicator is fused, and based on the fused evaluation result, a fusion indicator to be recommended is generated.

[0032] In some embodiments, based on the evaluation result corresponding to each candidate indicator and the fusion weight corresponding to each candidate indicator, the evaluation result corresponding to each candidate indicator is fused, and based on the fused evaluation result, a fusion indicator to be recommended is generated, including:

[0033] Based on the evaluation result corresponding to each candidate indicator and the multiple groups of reference fusion weights corresponding to the multiple candidate indicators, the evaluation result corresponding to each candidate indicator is fused multiple times, and based on the multiple evaluation results obtained after the multiple fusions, multiple candidate fusion indicators are generated, wherein the group of reference fusion weights corresponding to the multiple candidate indicators includes the reference fusion weight corresponding to each candidate indicator;

[0034] Based on multiple candidate fusion indicators, reference time series, and multiple annotated prediction time series, obtaining first recommended reference information of each candidate fusion indicator, wherein the first recommended reference information of the candidate fusion indicator indicates consistency between an evaluation result obtained by performing a priority evaluation on a time series prediction method corresponding to the multiple annotated prediction time series based on the candidate fusion indicator and the annotated information;

[0035] Based on the first recommendation reference information of each candidate fusion indicator, a fusion indicator to be recommended is determined from a plurality of candidate fusion indicators.

[0036] Through the above method, based on multiple groups of reference fusion weights corresponding to multiple candidate indicators, multiple candidate fusion indicators are generated, and the fusion indicator to be recommended is selected from them. In other words, it can also be understood as a method of searching for at least one group of reference fusion weights from multiple groups of reference fusion weights, so that the fusion indicator obtained based on this group of reference fusion weights can more accurately and comprehensively measure the prediction effect of the time series prediction task.

[0037] In some embodiments, the method also includes: obtaining at least one time series feature based on the real time series of the time series prediction task, the at least one time series feature indicating at least one of the following: periodicity, stationarity, number of zero values, dimensional difference, and number of outliers of the time series of events; and determining the indicator to be recommended based on the at least one time series feature.

[0038] In some embodiments, determining the indicator to be recommended based on at least one time series feature includes:

[0039] Based on at least one time series feature and multiple indicators of the time series prediction task, determine second recommended reference information for each indicator, where the second recommended reference information of the indicator indicates the adaptation between the time series of the event and the indicator;

[0040] Based on the second recommendation reference information of each indicator, the indicator to be recommended is determined.

[0041] Through the above method, since different time series features can reflect the data characteristics of the time series from different dimensions, the second recommended reference information of the indicator is obtained based on the time series features, so that the second recommended reference information can comprehensively reflect from multiple dimensions whether this indicator is suitable for the time series processed by the time series prediction task, and provide technical support for the subsequent determination of the indicator to be recommended.

[0042] In a second aspect, an embodiment of the present application provides an indicator recommendation device for a time series prediction task, the device comprising at least one functional module for executing an indicator recommendation method for a time series prediction task as provided in the aforementioned first aspect or any possible implementation of the first aspect.

[0043] In a third aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs an indicator recommendation method for a timing prediction task as provided in the aforementioned first aspect or any possible implementation of the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the index recommendation method for the timing prediction task provided in the first aspect or any possible implementation of the first aspect. The storage medium includes but is not limited to volatile memory, such as random access memory, non-volatile memory, such as flash memory, hard disk drive (HDD), solid state drive (SSD).

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product including instructions, which, when executed on a computing device cluster, enables the computing device cluster to implement the index recommendation method for a timing prediction task provided by the aforementioned first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when it is necessary to implement the index recommendation method for the aforementioned timing prediction task, the computer program product may be downloaded and executed on a computing device or a computing device cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of a connection method of a computing device cluster provided in an embodiment of the present application;

[0050] Figure 5 It is a flowchart of an indicator recommendation method for a time series prediction task provided by an embodiment of the present application;

[0051] Figure 6 This is a flow chart of determining an indicator to be recommended provided by an embodiment of the present application;

[0052] Figure 7 It is a flowchart of another indicator recommendation method for a time series prediction task provided by an embodiment of the present application;

[0053] Figure 8 is a schematic diagram of an indicator recommendation based on a first recommendation method provided in an embodiment of the present application;

[0054] Fig. 9 is a schematic diagram of an indicator recommendation based on a second recommendation method provided in an embodiment of the present application;

[0055] Fig.10 is a schematic diagram of an indicator recommendation based on the third recommendation method provided in an embodiment of the present application;

[0056] Fig.11 It is a schematic diagram of an indicator recommendation method for a time series prediction task provided by an embodiment of the present application;

[0057] Fig.12 It is a schematic diagram of an indicator recommendation device for a time series prediction task provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the relevant time series of the time series prediction task involved in this application, multiple indicators of the time series prediction task, etc. are all obtained with full authorization.

[0059] For ease of understanding, the key terms and key concepts involved in this application are explained below.

[0060] A time series is a sequence of values ​​of the same statistical dimension arranged in the order of their occurrence. In some embodiments, a time series is a set of random variables ordered by time, which is usually the result of observing a certain underlying process of an event at a given sampling rate in equally spaced time periods.

[0061] Time series forecasting, or time series forecasting for short, is a way to predict future time series based on historical time series of events. It should be understood that time series essentially reflects the trend of one or more random variables changing over time. The core of time series forecasting is to mine this law from historical time series and use it to predict future time series.

[0062] A time series prediction model, referred to as a time series prediction model, is a model that infers future numerical changes of a research variable based on its observed value and its change pattern. In an embodiment of the present application, a computer task performed by applying a time series prediction model is referred to as a time series prediction task. Different time series prediction models can be understood as different time series prediction methods, and the same time series prediction model using different model parameters can also be understood as different time series prediction methods. Schematically, a time series prediction model is, for example, an autoregressive model (AR), a moving average model (MA), an autoregressive and moving average model (ARMA), an autoregressive integrate moving average model (ARIMA), a seasonal autoregressive moving average model (SARIMA), and the like, but the present application is not limited thereto.

[0063] Indicators, or evaluation indicators, in this application refer to indicators for time series prediction tasks, which are used to measure the prediction effect of time series prediction tasks, or to evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks. In an embodiment of the present application, there may be multiple indicators for time series prediction tasks. For example, multiple indicators of time series prediction tasks include: mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), mean absolute percentage error (MAPE), coefficient of determination (or goodness of fit), weighted quantile loss (wQL), average weighted quantile loss (average wQL), weighted absolute percentage error (WAPE), mean absolute scaled error (MASE), symmetric mean absolute percentage error (SMAPE), etc., and this application does not limit this.

[0064] The autocorrelation coefficient is a statistic used to measure the correlation between a time series and itself at different time lags. For a given time series, the autocorrelation coefficient can reveal the dependency of data at different time lags.

[0065] ADF test (augmented Dicker-Fuller test) is a commonly used unit root test method, which determines the stationarity of time series by constructing an extended Dickey-Fuller regression model.

[0066] The application scenarios and implementation environment of this application are introduced below.

[0067] The technical solution provided by the present application can be applied to the scenario of evaluating the quality of time series forecasting tasks. Since the evaluation process of time series forecasting tasks often involves multiple indicators, different indicators are applicable to different types of time series. For example, MASE is applicable to time series with periodic or seasonal characteristics; MAPE is applicable to time series with significant differences in values ​​between different time points and significant influence of outliers; RMSE is more sensitive to outliers, that is, it is applicable to time series with outliers, and so on. Based on this, the present application provides an indicator recommendation method for time series forecasting tasks, which can automatically recommend indicators applicable to the time series forecasting task, or the most suitable indicators for the time series forecasting task, based on the relevant time series of the time series forecasting task, so as to improve the expected compliance of the evaluation results of the time series forecasting task. Among them, time series forecasting tasks are, for example, material demand forecasting, cost forecasting, temperature forecasting, power load forecasting, etc. The present application does not limit the type of time series forecasting tasks.

[0068] Figure 1 Schematic diagram of an implementation environment provided by the embodiment of the present application. Figure 1 As shown, the implementation environment includes a terminal 101 and a server 102, and the terminal 101 is directly or indirectly connected to the server 102 via a wireless network or a wired network.

[0069] An application is installed and running on the terminal 101, and the application has an evaluation function for the time series prediction task. For example, the application is a browser, a small program, a client, etc., but the present application is not limited to this. Schematically, the terminal 101 displays the application interface of the application, responds to various trigger operations performed by the user on the application interface, obtains the relevant time series of the time series prediction task, displays the recommended indicators for the time series prediction task, and displays the evaluation results of the time series prediction task obtained based on the indicators, etc., but the present application is not limited to this. Schematically, the terminal 101 is at least one of a smart phone, a game console, a desktop computer, an augmented reality terminal, a tablet computer, an e-book reader, and a laptop computer, but the present application is not limited to this.

[0070] The server 102 is used to provide background services for the application running on the terminal 101, for example, based on the relevant time series of the time series prediction task, determine the index to be recommended, and for example, based on the relevant time series of the index and the time series prediction task, evaluate the time series prediction task to obtain the corresponding evaluation result, etc., which is not limited in this application. Schematically, the server 102 is an independent physical server, or a server cluster or distributed file system composed of multiple physical servers.

[0071] In some embodiments, the terminal 101 and the server 102 work together. For example, the terminal 101 sends the acquired relevant time series of the time series prediction task to the server 102, and the server 102 determines the indicator to be recommended based on the relevant time series of the time series prediction task, and feeds back the indicator to be recommended to the terminal 101. This application does not limit this.

[0072] In some embodiments, the server 102 is deployed on a cloud platform, and the index recommendation method for the time series prediction task provided in this application can be performed by a cloud platform, wherein the cloud platform is the abbreviation of a cloud computing platform, which refers to a service based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", the huge data computing process is processed and analyzed at the remote end and returned to the user, which has the characteristics of large-scale, distributed, virtualized, high availability, scalability, on-demand service and security. The cloud platform can realize the rapid issuance and release of configurable computing resources with a small management cost, or a low interaction complexity between the user and the service provider. Schematically, the cloud platform is a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In some embodiments, the cloud platform can also be a virtual machine instance, a container instance, etc., and this application is not limited thereto. Schematically, the terminal 101 installs and runs the application according to the instructions of the cloud platform, responds to various trigger operations implemented by the user on the application interface according to the instructions of the cloud platform, obtains the relevant time series of the time series prediction task, displays the recommended indicators for the time series prediction task, and displays the evaluation results of the time series prediction task based on the indicators, etc. That is, when the method provided in the present application is executed by the cloud platform, for the terminal 101 used by the user, the actions performed by the terminal 101 are supported by the cloud platform.

[0073] It should be understood that the terminal 101 may refer to one of multiple terminals or a set of multiple terminals; the server 102 may refer to one of multiple servers or a set of multiple servers. The embodiment of the present application does not limit the number and type of each device in the implementation environment.

[0074] In some embodiments, the wireless network or wired network described above uses standard communication technology and / or protocol. The network includes, but is not limited to, any combination of a data center network, a storage area network (SAN), a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or a virtual private network. In some implementations, the data exchanged through the network is represented by technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. In addition, conventional encryption technologies such as secure socketslayer (SSL), transport layer security (TLS), virtual private network (VPN), and Internet protocol security (IPsec) can be used to encrypt all or part of the links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0075] The hardware structure of the terminal and server in the above implementation environment is introduced below.

[0076] The present application embodiment provides a computing device that can be configured as the above-mentioned server. Figure 2 , Figure 2 Schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application. Figure 2 As shown, the computing device 200 includes a memory 201, a processor 202, a communication interface 203 and a bus 204. The memory 201, the processor 202 and the communication interface 203 are connected to each other through the bus 204.

[0077] The memory 201 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. Schematically, the memory 201 is used to store at least one section of program code, and when the program code stored in the memory 201 is executed by the processor 202, the processor 202 is used to execute the steps performed by the computing device in the following method embodiment.

[0078] The processor 202 may be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit for controlling the execution of the program of the present application. The processor 202 may be a single-CPU processor or a multi-CPU processor. The number of processors 202 may be one or more. The communication interface 203 uses a transceiver module such as a transceiver to implement communication between the computing device 200 and other devices or communication networks. For example, data may be acquired through the communication interface 203.

[0079] The memory 201 and the processor 202 may be separately provided or integrated together.

[0080] Bus 204 may include a pathway for transmitting information between various components of computing device 200 (eg, memory 201 , processor 202 , communication interface 203 ).

[0081] The present application also provides a computing device cluster, which includes a plurality of computing devices. Figure 3 , Figure 3Schematic diagram of a computing device cluster provided in an embodiment of the present application. Figure 3 As shown, the computing device cluster includes multiple computing devices 200. The memory 201 in the multiple computing devices 200 in the computing device cluster may store the same instructions for executing the indicator recommendation method for the timing prediction task. In some embodiments, the memory 201 of the multiple computing devices 200 in the computing device cluster may also respectively store partial instructions for executing the indicator recommendation method for the timing prediction task. In other words, the combination of multiple computing devices 200 is used to jointly execute the instructions of the indicator recommendation method for the timing prediction task. Schematically, the computing device cluster can implement the indicator recommendation method for the timing prediction task provided in the present application.

[0082] In some embodiments, multiple computing devices 200 in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 4 Schematic diagram of a connection method of a computing device cluster provided in an embodiment of the present application. Figure 4 As shown, two computing devices 200 are connected via a network. Specifically, the network is connected via the communication interface 203 in each computing device 200. It should be understood that Figure 4 The functionality of the computing device 200 shown in FIG. 2 may also be performed by multiple computing devices 200 .

[0083] The following is an introduction to the indicator recommendation method for the time series prediction task provided by this application.

[0084] Figure 5 is a flowchart of an indicator recommendation method for a time series prediction task provided by an embodiment of the present application, such as Figure 5 As shown, this method is based on the above Figure 1 The server execution in the implementation environment is taken as an example for description. Schematically, the method includes the following steps 501 to 503.

[0085] 501. The server obtains a reference time series of a time series prediction task, where the time series prediction task indicates to predict a time series of events.

[0086] In the embodiments of the present application, a time series prediction task refers to a computer task performed by applying a time series prediction model. Please refer to the above description and will not be repeated here. In addition, the present application does not limit the type of event predicted by the time series prediction task, which may be material demand, cost, temperature, power load, etc. For any time series prediction task, the reference time series of the time series prediction task may be a real time series or a labeled time series, which is not limited in the present application. In some embodiments, the server obtains a reference time series uploaded by the user. Schematically, refer to the above description. Figure 1An application is installed and running on the terminal. The application has an evaluation function for time series prediction tasks. The server is used to provide background services for the application. The terminal displays the application interface of the application. In response to the user's upload operation on the application interface for the reference time series, the reference time series is sent to the server.

[0087] Schematically, a time series forecasting task indicates predicting the time series of an event in a second time period based on the time series of an event in a first time period, wherein the first time period is before the second time period. For example, taking the event as material demand, the time series forecasting task indicates predicting the time series corresponding to the shipments of material A from April to June of XX based on the time series corresponding to the actual shipments of material A from January to March of XX. Based on this, the reference time series of the time series forecasting task is also the real time series corresponding to the shipments of material A from April to June of XX. The time series obtained by executing the time series forecasting task using the time series forecasting method is called the predicted time series, that is, the predicted time series corresponding to the shipments of material A from April to June of XX. It should be understood that this is only an example and does not constitute a limitation of the present application.

[0088] 502. The server obtains multiple annotated prediction time series of a time series prediction task, where the reference time series and the annotated prediction time series have an overlapping time period, and the time series prediction methods corresponding to any two annotated prediction time series are different. The multiple annotated prediction time series include annotation information, and the annotation information indicates the priority of the time series prediction method corresponding to each annotated prediction time series.

[0089] In an embodiment of the present application, multiple annotated prediction time series are obtained by annotating multiple prediction time series, and multiple prediction time series refer to time series obtained by using multiple different time series prediction methods to perform time series prediction tasks, that is, the time series prediction methods corresponding to any two annotated prediction time series are different. For any annotated prediction time series, there is an overlapping time period between the annotated prediction time series and the reference time series, wherein the overlapping time period may be a partial time period of the reference time series or the entire time period of the reference time series. For example, taking the event as material demand as an example, the reference time series is the real time series corresponding to the shipment volume of material A from April to June of XX, the annotated prediction time series is the prediction time series corresponding to the shipment volume of material A from April to June of XX, or the annotated prediction time series is the prediction time series corresponding to the shipment volume of material A from May to June of XX, etc., and the present application does not limit this.

[0090] Among them, labeling multiple prediction time series means labeling the accuracy of the prediction time series based on the consistency between the prediction time series and the reference time series. For example, the labeling information is set to a labeling score, such as 0-10 points. The higher the score, the higher the consistency between the prediction time series and the reference time series, that is, the higher the accuracy of the prediction time series. Correspondingly, the higher the priority of the corresponding time series prediction method is, and the more it meets the business needs.

[0091] In some embodiments, the server obtains multiple annotated prediction time series uploaded by the user. This process is the same as the process of obtaining the reference time series in the aforementioned step 501, so it will not be repeated. In other embodiments, the server obtains the real time series of the time series prediction task uploaded by the user, and based on the real time series, uses multiple different time series prediction methods to perform the time series prediction task to obtain multiple prediction time series. Based on the multiple prediction time series, the annotation information is obtained to obtain multiple annotated prediction time series. That is, the server can generate multiple prediction time series based on the real time series uploaded by the user, and feed these prediction time series back to the user, and the user will annotate and obtain the annotation information. In this process, the server can determine part of the prediction time series from the generated multiple prediction time series to feed back to the user to improve processing efficiency, or it can feed all of them back to the user to improve the comprehensiveness of the annotation information. This application is not limited to this.

[0092] 503. The server determines the index to be recommended based on the reference time series and the multiple annotated prediction time series, where the index is used to evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods.

[0093] In the embodiment of the present application, the indicator is used to evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks, which means that based on the indicator and the relevant time series of the time series prediction task, the prediction effect when different time series prediction methods are used to perform the time series prediction task is evaluated to obtain the evaluation result, and the evaluation result is used to reflect the accuracy of the prediction time series obtained by using different time series prediction methods to perform the time series prediction task, thereby measuring the prediction effect. Taking the mean absolute error MAE as an example, the index value of MAE is obtained by calculating the absolute value of the difference between the predicted value and the true value of each sample, such as by calculating the absolute value of the difference between the predicted shipment volume and the actual shipment volume of material A at each time point from April to June in XX. It should be understood that the index value of MAE is also the evaluation result of the time series prediction task obtained based on MAE. Generally, the larger the index value of MAE, the lower the accuracy of the predicted time series. Correspondingly, the prediction effect of the time series prediction method corresponding to the predicted time series is worse, the priority is lower, and the less it meets the business needs.

[0094] The indicator to be recommended can be an indicator selected by the server from multiple indicators for the time series prediction task based on a reference time series and multiple annotated predicted time series, or it can be a fusion indicator obtained by fusing multiple indicators. Moreover, the present application does not limit the number of indicators to be recommended, which can be one or more. Among them, the multiple indicators for the time series prediction task refer to the above introduction, which will not be repeated here. The indicators of the time series prediction task can also be custom indicators, but the present application is not limited to this. Schematically, an indicator database is configured on the server. When the server obtains the relevant time series of the time series prediction task, it obtains multiple indicators from the indicator database. For example, it can be all indicators in the indicator database, or it can be multiple indicators related to the time series prediction task. The present application does not limit this.

[0095] The following is an introduction to the process of determining the recommended indicators, which includes the following steps:

[0096] Step 5031: Determine first recommended reference information for each indicator based on the reference time series, multiple annotated prediction time series, and multiple indicators of the time series prediction task.

[0097] Among them, for any indicator, the first recommended reference information of the indicator indicates the consistency between the evaluation result obtained by priority evaluation of the time series prediction methods corresponding to multiple annotated prediction time series based on the indicator and the annotated information. The following takes any indicator (hereinafter referred to as the first indicator) as an example to introduce the method for determining the first recommended reference information, including the following steps:

[0098] Step A1: Based on a reference time series, a plurality of annotated prediction time series and a first indicator, an evaluation result corresponding to the first indicator is obtained.

[0099] Among them, the server evaluates multiple annotated prediction time series based on the first indicator and the reference time series, reflects the accuracy of the prediction time series through the evaluation results, and then measures the prediction effect of the time series prediction task. Taking the first indicator as the mean absolute error MAE as an example, the evaluation result corresponding to the first indicator is also the indicator value of MAE, which will not be repeated here. For example, multiple annotated prediction time series include prediction time series 1 obtained by time series prediction method 1 and prediction time series 2 obtained by time series prediction method 2. Based on the indicator MAE and the reference time series, these two prediction time series are evaluated to obtain MAE indicator value 1 and MAE indicator value 2 respectively, that is, the evaluation result corresponding to the first indicator is obtained. If the indicator value 1 is greater than the indicator value 2, it indicates that the accuracy of the prediction time series 1 is lower than the accuracy of the prediction time series 2. Accordingly, the priority of the time series prediction method 1 is lower than the priority of the time series prediction method 2.

[0100] Step A2: based on the evaluation result and the annotation information corresponding to the first indicator, obtain a recommended score value of the first indicator, where the recommended score value of the first indicator indicates the consistency between the evaluation result and the annotation information corresponding to the first indicator.

[0101] Among them, since the evaluation result and annotation information corresponding to the first indicator can both indicate the priority of different time series prediction methods, but the annotation information is obtained through annotation, it is more in line with user needs. Therefore, in this step, by obtaining the recommended score value of the first indicator, the consistency between the evaluation result and the annotation information corresponding to the first indicator can be reflected. For example, the evaluation result corresponding to the first indicator indicates that the priority of time series prediction method 1 is lower than the priority of time series prediction method 2, and the annotation information also indicates that the priority of time series prediction method 1 is lower than the priority of time series prediction method 2, which indicates that the first indicator can accurately evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks. Schematically, the recommended score value of the first indicator can be calculated using normalized discounted cumulative gain (NDCG), or other methods can be used, which is not limited in this application. Among them, NDCG can be used to compare different results between [0, 1]. Generally, the larger the NDCG value, the stronger the consistency between the evaluation result corresponding to the first indicator and the labeled evaluation result corresponding to the first indicator, and the first indicator can more accurately evaluate the prediction effect when using different time series prediction methods to perform time series prediction tasks, or measure the prediction effect of time series prediction tasks.

[0102] Step A3: Determine first recommended reference information for the first indicator based on the recommendation score value of the first indicator.

[0103] The server may use the recommendation score value as the first recommendation reference information of the first indicator, or process the recommendation score value and convert it into text type information to obtain the first recommendation reference information. That is, the present application does not limit the data type of the first recommendation reference information, which may be a numerical value, such as an integer, a percentage, etc., or a text, etc. When the data type of the first recommendation reference information is a numerical value, the first recommendation reference information may also be understood as a first score value for the first indicator.

[0104] Through the above steps A1 to A3, taking the first indicator as an example, the method for obtaining the first recommended reference information is introduced. For any indicator of the time series prediction task, the above steps A1 to A3 are executed to obtain the first recommended reference information of each indicator.

[0105] It should be understood that since the annotation information can accurately reflect the accuracy of the predicted time series, the first recommended reference information of the indicator is obtained based on the recommended score value between the evaluation result corresponding to the indicator and the annotation information, so that the first recommended reference information can reflect whether the prediction effect of the time series prediction task can be accurately measured through this indicator based on the consistency between the evaluation result corresponding to the indicator and the annotation information, thereby providing technical support for the subsequent determination of the indicator to be recommended.

[0106] Step 5032: Determine the indicator to be recommended based on the first recommendation reference information of each indicator.

[0107] Schematically, this step includes the following implementation methods:

[0108] Method 1: Based on the first recommendation reference information of each indicator, determine the indicator to be recommended from multiple indicators.

[0109] The server sorts the multiple indicators based on the first recommended reference information of each indicator, and uses the indicators ranked in the top N as the indicators to be recommended. N is a positive integer and can be set according to actual needs, such as specified by the user. For example, N is 3, that is, based on the first recommended reference information of each indicator, 3 indicators to be recommended are selected from the multiple indicators.

[0110] Through the above-mentioned method one, it is possible to conveniently and quickly select indicators suitable for time series prediction tasks from multiple indicators based on the first recommended reference information of each indicator, and use them as indicators to be recommended. This can improve the processing efficiency of the server on the basis of improving the expected conformity of the evaluation results of the time series prediction task.

[0111] Method 2: Based on the first recommendation reference information of each indicator, multiple candidate indicators are determined from multiple indicators, the evaluation results corresponding to each candidate indicator are integrated, and based on the integrated evaluation results, a fused indicator to be recommended is generated.

[0112] Among them, the fusion index is used to comprehensively evaluate the prediction effect of using different time series prediction methods to perform time series prediction tasks from multiple evaluation dimensions. The implementation process of this method is introduced below, including the following steps:

[0113] Step B1: determining a plurality of candidate indicators from a plurality of indicators based on the first recommended reference information of each indicator.

[0114] The implementation of step B1 can refer to the aforementioned method 1. That is, the server can sort multiple indicators and select multiple candidate indicators from the multiple indicators. The number of candidate indicators can be set according to business needs, such as specified by the user. For example, the number of candidate indicators is set to 3, but the present application is not limited thereto.

[0115] Step B2: Based on the evaluation result corresponding to each candidate indicator and the fusion weight corresponding to each candidate indicator, the evaluation result corresponding to each candidate indicator is fused, and based on the fused evaluation result, a fusion indicator to be recommended is generated.

[0116] Among them, the fusion weight corresponding to each candidate indicator is a preset weight, which can be set according to business needs, and this application does not limit this. In some embodiments, the server obtains the evaluation results corresponding to each candidate indicator, and based on the fusion weight corresponding to each candidate indicator, the evaluation results corresponding to each candidate indicator are weighted and summed to obtain the fused evaluation results, and based on the fused evaluation results, the fusion indicator to be recommended is generated. This process can also be understood as fusing the evaluation results of multiple candidate indicators to generate a fusion indicator for the fused evaluation results. For example, taking multiple candidate indicators as MAE, MSE and RMSE as an example, the server obtains the evaluation results of these candidate indicators: the index value of MAE, the index value of MSE and the index value of RMSE, based on the fusion weight corresponding to each candidate indicator, the index value of each candidate indicator is weighted and summed to obtain the fused index value, and based on the fused index value, the fusion indicator "MAE+MSE+RMSE" is generated (this is only an example and does not constitute a limitation of this application).

[0117] In some other embodiments, the fusion weight corresponding to each candidate indicator is obtained by searching multiple groups of reference fusion weights. Based on this, step B2 includes the following sub-steps:

[0118] Step B21: Based on the evaluation result corresponding to each candidate indicator and multiple groups of reference fusion weights corresponding to multiple candidate indicators, the evaluation result corresponding to each candidate indicator is fused multiple times, and multiple candidate fusion indicators are generated based on the multiple evaluation results obtained after the multiple fusions.

[0119] Among them, a group of reference fusion weights corresponding to multiple candidate indicators includes a reference fusion weight corresponding to each candidate indicator. For example, taking multiple candidate indicators as MAE, MSE and RMSE as an example, a group of reference fusion weights includes three reference fusion weights, namely, the reference fusion weight Y1 corresponding to MAE, the reference fusion weight Y2 corresponding to MSE, and the reference fusion weight Y3 corresponding to RMSE, Y1+Y2+Y3=1, schematically, Y1, Y2, Y3∈[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], based on this, multiple groups of reference fusion weights can be set, for example, the first group of reference fusion weights is Y1=0.1, Y2=0.2, Y3=0.7; the second group of reference fusion weights is Y1=0.2, Y2=0.4, Y3=0.4; etc. It should be understood that this is only an example, and the present application does not limit the number of multiple groups of reference fusion weights, weight values, etc., and can be set according to business needs.

[0120] In this step B21, for any set of reference fusion weights, the server fuses the evaluation results corresponding to each candidate indicator based on the set of reference fusion weights (for example, by weighted summation), obtains the fused evaluation results, and generates candidate fusion indicators based on the fused evaluation results. For example, taking the number of candidate indicators as 3 as an example, the candidate fusion indicators are generated by the following formula (1):

[0121]

[0122] In the above formula (1), i is a positive integer, indicating any one, indicator i represents the index value of the i-th candidate index, Y i It should be noted that the above method of generating candidate fusion indicators is only an example and does not constitute a limitation on the present application. For example, the candidate fusion indicators can also be generated by weighted summing and then averaging, etc.

[0123] For each set of reference fusion weights, the above method is executed to fuse the evaluation results corresponding to each candidate indicator multiple times, and multiple candidate fusion indicators are generated based on the multiple evaluation results obtained after multiple fusions. That is, a set of reference fusion weights corresponds to one fusion to generate one candidate fusion indicator.

[0124] Step B22: Based on multiple candidate fusion indicators, reference time series, and multiple annotated prediction time series, obtain the first recommended reference information of each candidate fusion indicator, wherein the first recommended reference information of the candidate fusion indicator indicates the consistency between the evaluation result obtained by performing a priority evaluation on the time series prediction method corresponding to the multiple annotated prediction time series based on the candidate fusion indicator and the annotated information.

[0125] Among them, for any candidate fusion indicator, the server obtains the first recommended reference information of the candidate indicator based on the candidate indicator, the reference time series and multiple annotated prediction time series. This process is the same as the aforementioned step 5031, so it is not repeated here. In other words, step B22 is to obtain the first recommended reference information for each candidate fusion indicator, so as to reflect whether the candidate fusion indicator can accurately evaluate the prediction effect when performing the time series prediction task using different time series prediction methods, or measure the prediction effect of the time series prediction task, so as to provide technical support for the subsequent determination of the fusion indicator to be recommended.

[0126] Step B23: Determine a fusion indicator to be recommended from multiple candidate fusion indicators based on the first recommendation reference information of each candidate fusion indicator.

[0127] The implementation method of step B23 can refer to the aforementioned method 1. That is, the server can sort multiple candidate fusion indicators and select the fusion indicator to be recommended from the multiple candidate fusion indicators. The number of fusion indicators to be recommended can be one or more, and this application does not limit this.

[0128] After the above steps B21 to B23, multiple candidate fusion indicators are generated based on multiple groups of reference fusion weights corresponding to multiple candidate indicators, and the fusion indicator to be recommended is selected therefrom. In other words, it can also be understood as a method of searching for at least one group of reference fusion weights from multiple groups of reference fusion weights, so that the fusion indicator obtained based on the group of reference fusion weights can more accurately and comprehensively evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks, or measure the prediction effect of time series prediction tasks.

[0129] Through the above-mentioned method 2, the fusion index generated based on multiple candidate indicators can comprehensively evaluate the prediction effect when using different time series prediction methods to perform time series prediction tasks from multiple evaluation dimensions, or measure the prediction effect of the time series prediction task, thereby effectively improving the expected conformity of the evaluation results of the time series prediction task.

[0130] In some embodiments, the above-mentioned method one and method two can also be combined. Schematically, based on the first recommended reference information of each indicator, multiple indicators are sorted, and the indicators ranked in the top K positions are determined as target indicators (K is a positive integer, for example, set to 1). If the target indicator meets the preset conditions (for example, the first recommended reference information is greater than or equal to the preset threshold), the target indicator is used as the indicator to be recommended. If the target indicator does not meet the preset conditions, the implementation process shown in method two is executed to generate the fusion indicator to be recommended.

[0131] Schematically, refer to Figure 6 , Figure 6 FIG. 1 is a flow chart of determining an indicator to be recommended provided by an embodiment of the present application. Figure 6 As shown, taking K as 1 and the first recommended reference information as a numerical value as an example, the server sets a preset threshold value T. If the first recommended reference information of the target indicator is greater than or equal to T, the target indicator is fed back to the user as the indicator to be recommended. If the first recommended reference information of the target indicator is less than T, multiple candidate indicators (for example, 3) are selected from multiple indicators, and multiple candidate fusion indicators are generated based on multiple groups of reference fusion weights of multiple candidate indicators. The fusion indicator to be recommended is determined and fed back to the user. In this way, it is possible to automatically determine whether the indicator to be recommended is directly selected from multiple indicators or to generate a fusion indicator based on the first recommended reference information of the indicator. On the basis of improving the expected conformity of the evaluation results of the time series prediction task, it is possible to ensure that the indicator to be recommended can accurately evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods, or measure the prediction effect of the time series prediction task.

[0132] In other embodiments, the above-mentioned method 1 and method 2 may be combined to feed back the indicators to be recommended based on method 1 and the indicators to be recommended based on method 2 to the user, which is not limited in this application.

[0133] In addition, based on the above introduction, it can be known that for any annotated prediction time series, there is an overlapped time period between the annotated prediction time series and the reference time series. In some embodiments, the server determines the overlapped time period between the reference time series and each annotated prediction time series, and determines the index to be recommended based on the time series corresponding to the overlapped time period, that is, executing the above steps 5031 and 5032. The process is similar, so it will not be repeated. By determining the overlapped time period to further determine the index to be recommended, the accuracy of the determined index can be improved. For example, the reference time series is the real time series corresponding to the shipment volume of material A from April to June of XX, the marked forecast time series 1 is the forecast time series corresponding to the shipment volume of material A from April to June of XX, and the marked forecast time series 2 is the forecast time series corresponding to the shipment volume of material A from May to June of XX. When evaluating these two forecast time series based on the indicator MAE, for forecast time series 1, the evaluation is based on the time series of the overlapping time period from April to June, and for forecast time series 2, the evaluation is based on the time series of the overlapping time period from May to June.

[0134] In some embodiments, after determining the indicator to be recommended, the server sends the indicator to the terminal used by the user, and the terminal displays it. The server may also send the recommended reference information of the indicator to the terminal, which is displayed by the terminal to provide guidance information to the user, and this application is not limited to this. If the indicator is a fusion indicator, the server can send a prompt message for the fusion indicator to the terminal, which is displayed by the terminal. The prompt message indicates that the fusion indicator is used to comprehensively evaluate the prediction effect when performing the time series prediction task using different time series prediction methods from multiple evaluation dimensions. In this way, the purpose of reminding users is achieved and the efficiency of information access is improved.

[0135] In summary, in the indicator recommendation method for the time series prediction task provided in the present application, a reference time series of the time series prediction task and multiple annotated prediction time series are obtained, and the indicators to be recommended are determined based on the obtained time series, wherein the multiple annotated prediction time series include annotation information to indicate the priority of the time series prediction method corresponding to each annotated prediction time series. Therefore, the annotation information can accurately reflect the prediction effect of the time series prediction method corresponding to each annotated prediction time series. Furthermore, the indicator determined by using this annotation information can accurately evaluate the prediction effect when different time series prediction methods are used to perform the time series prediction task, thereby improving the expected conformity of the evaluation results for the time series prediction task.

[0136] After the above Figure 5The method embodiment shown introduces an indicator recommendation method for a time series prediction task provided by the present application. In some embodiments, the server can also determine the indicator to be recommended based on the characteristics of the time series of the event. For example, before executing the aforementioned step 501, the server first determines the indicator to be recommended based on the characteristics of the time series of the event, and feeds the indicator back to the user. If the user requests to re-recommend the indicator, the server executes the aforementioned steps 501 to 503 to re-determine the indicator to be recommended. The following is an introduction to the process of the server determining the indicator to be recommended based on the characteristics of the time series of the event, which includes the following steps:

[0137] Step C1: based on the real time series of the time series prediction task, obtain at least one time series feature.

[0138] Among them, at least one time series feature indicates at least one of the following: periodicity, stationarity, number of zero values, dimensional difference, and number of outliers of the time series of the event. Schematically, the server obtains the real time series uploaded by the user, performs feature extraction on the real time series, and obtains at least one time series feature. By extracting features from the real time series, the obtained time series features can accurately reflect the characteristics of the time series of the event. Of course, the server can also extract features from the predicted time series, or extract features from the predicted time series and the reference time series, which is not limited in this application.

[0139] Illustratively, the timing features are illustrated below with reference to Table 1. It should be understood that the timing features shown in the table are only for illustration and do not constitute a limitation on the present application. In some embodiments, corresponding timing features can be extracted according to business requirements, such as specified by the user.

[0140] Table 1

[0141] Feature Encoding Feature Name meaning <![CDATA[f1]]> Autocorrelation coefficient Periodicity of time series <![CDATA[f2]]> ADF test statistic Stationarity of time series <![CDATA[f3]]> Zero value ratio The number of zero values ​​in the time series <![CDATA[f4]]> Mean range between time series Dimensional differences in time series <![CDATA[f5]]> Outlier ratio The number of outliers in the time series

[0142] Step C2: Determine the indicator to be recommended based on at least one time series feature.

[0143] The index to be recommended may be an index selected by the server from multiple indexes for the time series prediction task based on at least one time series feature, or a fusion index obtained by fusing multiple indexes. Moreover, the present application does not limit the number of indexes to be recommended, which may be one or more. Schematically, this step C2 includes the following sub-steps:

[0144] Step C21: Based on at least one time series feature and multiple indicators of the time series prediction task, determine the second recommended reference information of each indicator, where the second recommended reference information of the indicator indicates the adaptation between the time series of the event and the indicator.

[0145] Among them, for any indicator (hereinafter referred to as the first indicator), the process of the server obtaining the second recommended reference information of the indicator includes: determining the second recommended reference information of the first indicator based on at least one timing feature and the weight of the first indicator corresponding to each timing feature, and the weight of the first indicator corresponding to each timing feature indicates the adaptation between the first indicator and each timing feature.

[0146] Among them, the weight of the first indicator corresponding to each timing feature is a preset weight, which can be set according to business needs, such as specified by the user, and this application does not limit this. Schematically, the server performs weighted summation on at least one timing feature based on the weight of the first indicator corresponding to each timing feature to obtain the second recommended reference information of the first indicator, which can also be understood as an evaluation result for the first indicator. For example, in combination with the aforementioned Table 1, the second recommended reference information of the first indicator is calculated by the following formula (2):

[0147]

[0148] In the above formula (2), i is a positive integer, representing any one, W i represents the weight of the first indicator corresponding to the i-th time series feature, f i Represents the i-th time series feature. Taking MAE as an example, the weights of MAE corresponding to each time series feature are: W mae 1. W mae 2. W mae 3. W mae 4. W mae 5. The second recommended reference information of MAE is calculated based on the following formula (3).

[0149]

[0150] It should be noted that the above-mentioned method for obtaining the second recommended reference information is only an example and does not constitute a limitation on the present application. For example, the second recommended reference information can also be obtained by weighted summing and then calculating the average value, etc. In addition, the present application does not limit the data type of the second recommended reference information, which can be a numerical value, such as an integer, a percentage, etc., or a text, etc. In the case where the data type of the second recommended reference information is a numerical value, the second recommended reference information can also be understood as a second score value for the first indicator.

[0151] For any indicator, the steps shown above are performed to obtain the second recommended reference information for each indicator. It should be understood that since different time series features can reflect the data characteristics of the time series from different dimensions, the second recommended reference information of the indicator is obtained based on the time series features, so that the second recommended reference information can comprehensively reflect from multiple dimensions whether this indicator is suitable for the time series processed by the time series prediction task, and provide technical support for the subsequent determination of the indicator to be recommended.

[0152] Step C22: Determine the indicator to be recommended based on the second recommendation reference information of each indicator.

[0153] Similar to the aforementioned step 5032, this step C22 also includes the following implementation methods:

[0154] Method 1: Based on the second recommendation reference information of each indicator, determine the indicator to be recommended from multiple indicators.

[0155] The server sorts the multiple indicators based on the second recommendation reference information of each indicator, and uses the indicators ranked in the top M as the indicators to be recommended. M is a positive integer and can be set according to actual needs. For example, M is 3, that is, based on the second recommendation reference information of each indicator, 3 indicators to be recommended are selected from the multiple indicators.

[0156] Through the above-mentioned method one, it is possible to conveniently and quickly select indicators suitable for time series prediction tasks from multiple indicators based on the second recommended reference information of each indicator, and use them as indicators to be recommended. This can improve the processing efficiency of the server on the basis of improving the expected conformity of the evaluation results of the time series prediction task.

[0157] Method 2: Based on the second recommendation reference information of each indicator, multiple candidate indicators are determined from multiple indicators, the evaluation results corresponding to each candidate indicator are fused, and based on the fused evaluation results, a fused indicator to be recommended is generated.

[0158] The implementation method of this process is the same as the second method in the aforementioned step 5032, so it will not be repeated here.

[0159] Through the above-mentioned method 2, the fusion index generated based on multiple candidate indicators can comprehensively evaluate the prediction effect when using different time series prediction methods to perform time series prediction tasks from multiple evaluation dimensions, or measure the prediction effect of the time series prediction task, thereby effectively improving the expected conformity of the evaluation results of the time series prediction task.

[0160] In other embodiments, the above-mentioned method 1 and method 2 may be combined to feed back the indicators to be recommended based on method 1 and the indicators to be recommended based on method 2 to the user, which is not limited in this application.

[0161] It should be understood that since the second recommended reference information of the indicator can indicate the adaptation between the time series of the event and the indicator, and the adaptation between the time series of the event and the indicator can reflect whether this indicator is suitable for the time series processed by the time series prediction task, therefore, through the second recommended reference information of each indicator, the indicator that is most suitable for the time series prediction task can be selected from multiple indicators, and then the selected indicator can be recommended to the user, or multiple indicators can be fused to obtain a fused indicator to comprehensively evaluate the prediction effect when using different time series prediction methods to perform the time series prediction task, or to measure the prediction effect of the time series prediction task, thereby improving the expected conformity of the evaluation result of the time series prediction task.

[0162] In other embodiments, the server can also determine the indicator to be recommended in combination with the aforementioned first recommendation reference information and the second recommendation reference information. For example, after executing the aforementioned steps 501 and 502, the server determines the first recommendation reference information and the second recommendation reference information for each indicator based on the reference time series, multiple annotated prediction time series, the real time series, and multiple indicators of the time series prediction task, and determines the indicator to be recommended based on the first recommendation reference information and the second recommendation reference information of each indicator. It should be understood that since the first recommendation reference information and the second recommendation reference information can respectively indicate: the adaptation between the time series of the event and the indicator, and the consistency between the evaluation result obtained by priority evaluation of the time series prediction method corresponding to the multiple annotated prediction time series based on the indicator and the annotation information. Among them, the adaptation between the time series of events and the indicator can reflect whether this indicator is suitable for the time series processed by the time series prediction task. The consistency between the evaluation results obtained by priority evaluation of the time series prediction methods corresponding to multiple annotated prediction time series based on this indicator and the annotation information can comprehensively reflect whether this indicator can accurately measure the prediction effect of the time series prediction task. Therefore, through a variety of recommended reference information for each indicator, the most suitable indicator for the time series prediction task can be selected from multiple indicators, and then the selected indicator can be recommended to the user, or multiple indicators can be fused to obtain a fused indicator to comprehensively measure the prediction effect of the time series prediction task, thereby improving the expected conformity of the evaluation results of the time series prediction task.

[0163] Illustratively, similar to the aforementioned step 5032 and step C2, the server determines the indicator to be recommended based on the first recommendation reference information and the second recommendation reference information of each indicator, including the following implementation methods:

[0164] Method 1: Based on the first recommendation reference information and the second recommendation reference information of each indicator, determine the indicator to be recommended from multiple indicators.

[0165] The server sorts the multiple indicators based on the first recommended reference information and the second recommended reference information of each indicator, and uses the indicators ranked in the top L as indicators to be recommended. For example, the server sums the first recommended reference information and the second recommended reference information of each indicator, and sorts the multiple indicators based on the summation result, but the present application is not limited to this. L is a positive integer and can be set according to actual needs. For example, L is 3, that is, based on the first recommended reference information and the second recommended reference information of each indicator, 3 indicators to be recommended are selected from the multiple indicators.

[0166] Method 2: Based on the first recommendation reference information and the second recommendation reference information of each indicator, multiple candidate indicators are determined from multiple indicators, the evaluation results corresponding to each candidate indicator are fused, and based on the fused evaluation results, a fused indicator to be recommended is generated.

[0167] The implementation method of this process is the same as the second method in the aforementioned step 5032, so it will not be repeated here.

[0168] In other embodiments, the above-mentioned method 1 and method 2 may be combined to feed back the indicators to be recommended based on method 1 and the indicators to be recommended based on method 2 to the user, which is not limited in this application.

[0169] After the foregoing introduction, the indicator recommendation method for the time series prediction task provided by the present application is introduced. Based on the foregoing content, it can be known that the recommended reference information of the indicator includes at least one of the first recommended reference information and the second recommended reference information. In some embodiments, the server can obtain the recommended reference information of the indicator according to the indicator recommendation method selected by the user, and make corresponding indicator recommendations. Figure 7 , this implementation method is introduced.

[0170] Figure 7 FIG. 1 is a flowchart of another method for recommending indicators for a time series prediction task provided by an embodiment of the present application. Figure 7 As shown, the method is described by taking the method executed by the server as an example. Schematically, the method includes the following steps 701 to 708.

[0171] 701. The server obtains a reference time series of a time series prediction task, a plurality of annotated prediction time series, and a real time series, wherein the time series prediction task indicates prediction of a time series of an event.

[0172] This step refers to the aforementioned steps 501, 502 and step C1 and will not be described in detail.

[0173] 702. The server determines a recommendation method for recommending indicators for the time series prediction task.

[0174] Among them, there are three ways to recommend indicators for time series prediction tasks:

[0175] (1) The first recommendation method, that is, recommending indicators for the time series prediction task based on the first recommended reference information of the indicators.

[0176] (2) The second recommendation method, that is, recommending indicators for the time series prediction task based on the second recommendation reference information of the indicators.

[0177] (3) A third recommendation method, namely, recommending an indicator based on the first recommendation reference information and the second recommendation reference information of the indicator.

[0178] In some embodiments, the terminal connected to the server is capable of displaying an application interface of an application, which provides an evaluation function for a timing prediction task, and in response to an evaluation operation on the timing prediction task, displays an indicator recommendation control for the timing prediction task; in response to a first trigger operation implemented on the indicator recommendation control, determines that the recommended method for recommending indicators for the timing prediction task is a first recommendation method, and notifies the server; or, in response to a second trigger operation implemented on the indicator recommendation control, determines that the recommended method for recommending indicators for the timing prediction task is a second recommendation method, and notifies the server; or, in response to a third trigger operation implemented on the indicator recommendation control, determines that the recommended method for recommending indicators for the timing prediction task is a third recommendation method, and notifies the server.

[0179] Illustratively, if the index recommendation is performed based on the first recommendation method, the server executes the following steps 703 and 704. If the index recommendation is performed based on the second recommendation method, the server executes the following steps 705 and 706. If the index recommendation is performed based on the third recommendation method, the server executes the following steps 707 and 708.

[0180] 703. The server determines first recommended reference information for each indicator based on the reference time series, multiple annotated prediction time series, and multiple indicators of the time series prediction task.

[0181] 704. The server determines the indicator to be recommended based on the first recommendation reference information of each indicator.

[0182] The implementation of step 703 and step 704 refers to the aforementioned step 503 and will not be described in detail.

[0183] Schematically, the indicator recommendation process shown in step 703 and step 704 can be referred to Figure 8 , Figure 8This is a schematic diagram of an indicator recommendation based on the first recommendation method provided by an embodiment of the present application. The server obtains a plurality of annotated prediction time series (the annotation information can be annotated manually or by machine, and the present application does not limit this); traverses each indicator, obtains the evaluation result (or indicator value) corresponding to each indicator, and obtains the recommended score value of each indicator based on the evaluation result and annotation information corresponding to each indicator (the consistency between the evaluation result and the annotation information is reflected by the recommended score value), that is, obtains the first recommended reference information of each indicator; based on the first recommended reference information of each indicator, sorts multiple indicators, and uses the indicators ranked in the top N as indicators to be recommended.

[0184] 705. The server obtains second recommendation reference information of each indicator based on the multiple indicators and the real time series.

[0185] 706. The server determines the indicator to be recommended based on the second recommendation reference information of each indicator.

[0186] The implementation of step 705 and step 706 refers to the aforementioned step C1 and step C2, and will not be described in detail.

[0187] Schematically, the indicator recommendation process shown in step 705 and step 706 can be referred to Fig. 9 , Fig. 9 is a schematic diagram of an indicator recommendation based on the second recommendation method provided in an embodiment of the present application. Fig. 9 As shown, Fig. 9 As shown, the server obtains at least one time series feature based on a real time series; determines the second recommended reference information of each indicator based on at least one time series feature and the weight of each indicator corresponding to each time series feature; and sorts multiple indicators based on the second recommended reference information of each indicator, and uses the indicators ranked in the top M positions as the indicators to be recommended.

[0188] 707. The server obtains first recommended reference information for each indicator and second recommended reference information for each indicator based on the reference time series, multiple annotated predicted time series, the real time series, and multiple indicators of the time series prediction task.

[0189] 708. The server determines the indicator to be recommended based on the first recommendation reference information of each indicator and the second recommendation reference information of each indicator.

[0190] The implementation of step 707 and step 708 is referred to the above description and will not be described in detail.

[0191] Schematically, the indicator recommendation process shown in step 707 and step 708 can be referred to Fig.10 , Fig.10is a schematic diagram of an indicator recommendation based on the third recommendation method provided in an embodiment of the present application. Fig.10 As shown, the server obtains the first recommended reference information and the second recommended reference information of each indicator respectively; sums the first recommended reference information and the second recommended reference information of each indicator, and sorts the multiple indicators based on the summation result (for example, weighted summation, average after weighted summation, etc.), and takes the indicators ranked in the top L as the indicators to be recommended. It should be understood that this is only an example and does not constitute a limitation of the present application. For example, three indicators can be selected based on the first recommended reference information, and three indicators can be selected based on the second recommended reference information. The intersection of these two parts of indicators is taken to obtain the indicators to be recommended, and so on.

[0192] In summary, in the indicator recommendation method for the time series prediction task provided in the present application, for multiple indicators of the time series prediction task, based on the indicator recommendation method determined by the user, the recommendation reference information of each indicator is obtained respectively, and the indicators to be recommended are determined based on the recommendation reference information of these indicators. In this way, it is possible to ensure that the recommended indicators can accurately evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods, or measure the prediction effect of the time series prediction task, thereby improving the expected conformity of the evaluation results of the time series prediction task while meeting the personalized needs of the user.

[0193] In some embodiments, based on the foregoing Figure 5 It can be seen from the embodiment shown that in the process of determining the indicator to be recommended, the two methods of "determining the indicator to be recommended from multiple indicators" and "generating a fusion indicator" can be combined. Similarly, the above Figure 7 The three recommendation methods involved in the method embodiment shown (ie, the three recommendation methods shown in steps 703 to 708) can also be implemented in combination with "generating fusion indicators". Fig.11 , an example is given to illustrate this process. Fig.11 is a schematic diagram of an indicator recommendation method for a time series prediction task provided by an embodiment of the present application. Fig.11 As shown in the figure, the indicator recommendation method for time series forecasting tasks includes the following steps:

[0194] Step 1: The server obtains multiple indicators of the time series prediction task, the reference time series, the annotated prediction time series, and the real time series. The multiple indicators are, for example, MAE, MSE, RMSE, MAPE, wQL, WAPE, MASE, SMAPE, etc., and can also be custom indicators, which are not limited in this application.

[0195] Step 2: The server determines a recommendation method for recommending indicators for the time series prediction task.

[0196] Step 3: If the first recommendation method is adopted, the server determines N indicators to be recommended from multiple indicators based on the first recommendation reference information of the indicators.

[0197] Step 4: If the second recommendation method is adopted, the server determines M indicators to be recommended from multiple indicators based on the second recommendation reference information of the indicators.

[0198] Step 5: If the third recommendation method is adopted, the server determines L indicators to be recommended from multiple indicators based on the first recommendation reference information and the second recommendation reference information of the indicators.

[0199] Step 6: For the indicators to be recommended determined in any one of the above steps 3 to 5, K target indicators are determined. If the target indicator meets the preset conditions (for example, the recommended reference information is greater than or equal to the preset threshold), the target indicator is fed back to the user as the indicator to be recommended. If the target indicator does not meet the preset conditions, a fusion indicator to be recommended is generated and fed back to the user.

[0200] In summary, in the indicator recommendation method for the time series prediction task provided in the present application, for multiple indicators of the time series prediction task, based on the indicator recommendation method determined by the user, the recommendation reference information of each indicator is obtained respectively, and according to the recommended reference information of the indicator, it is automatically determined whether the indicator to be recommended is directly selected from multiple indicators or a fusion indicator is generated. In this way, on the basis of meeting the personalized needs of users, the expected conformity of the evaluation results of the time series prediction task can be improved, ensuring that the indicator to be recommended can accurately evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods, or measure the prediction effect of the time series prediction task.

[0201] Fig.12 Schematic diagram of an indicator recommendation device for a time series prediction task provided by an embodiment of the present application. The device can be implemented as part or all of the functions of the aforementioned computing device through software, hardware, or a combination of both. Fig.12 As shown, the device includes a first acquisition module 1201 , a second acquisition module 1202 and a first determination module 1203 .

[0202] A first acquisition module 1201 is used to acquire a reference time series of a time series prediction task, where the time series prediction task indicates predicting a time series of an event;

[0203] The second acquisition module 1202 is used to acquire multiple annotated prediction time series of the time series prediction task, where the reference time series and the annotated prediction time series have an overlapping time period, and any two annotated prediction time series correspond to different time series prediction methods, and the multiple annotated prediction time series include annotation information, and the annotation information indicates the priority of the time series prediction method corresponding to each annotated prediction time series;

[0204] The first determination module 1203 is used to determine the index to be recommended based on the reference time series and multiple annotated prediction time series, where the index is used to evaluate the prediction effect when different time series prediction methods are used to perform the time series prediction task.

[0205] In some embodiments, the first acquisition module 1201 is used to: acquire a reference time series uploaded by a user.

[0206] In some embodiments, the second acquisition module 1202 is used for any of the following:

[0207] Get the predicted time series of multiple annotations uploaded by the user;

[0208] The real time series of the time series prediction task uploaded by the user is obtained, and based on the real time series, a plurality of different time series prediction methods are used to perform the time series prediction task to obtain a plurality of predicted time series, and based on the plurality of predicted time series, annotation information is obtained to obtain a plurality of annotated prediction time series.

[0209] In some embodiments, the first determining module 1203 is configured to:

[0210] Determine the time period of overlap between the reference time series and each annotated forecast time series;

[0211] Based on the time series corresponding to the overlapping time periods, the indicators to be recommended are determined.

[0212] In some embodiments, the first determining module 1203 includes:

[0213] A first determination unit is used to determine first recommended reference information for each indicator based on a reference time series, a plurality of annotated prediction time series, and a plurality of indicators of a time series prediction task, wherein the first recommended reference information of the indicator indicates consistency between an evaluation result obtained by performing a priority evaluation on a time series prediction method corresponding to a plurality of annotated prediction time series based on the indicator and the annotated information;

[0214] The second determining unit is used to determine the indicator to be recommended based on the first recommendation reference information of each indicator.

[0215] In some embodiments, the first determining unit is configured to:

[0216] Based on the reference time series, the multiple annotated prediction time series, and the first indicator, obtaining an evaluation result corresponding to the first indicator, where the first indicator refers to any one of the multiple indicators;

[0217] Based on the evaluation result and the annotation information corresponding to the first indicator, obtaining a recommended score value of the first indicator, the recommended score value of the first indicator indicating consistency between the evaluation result and the annotation information corresponding to the first indicator;

[0218] Based on the recommendation score value of the first indicator, first recommendation reference information of the first indicator is determined.

[0219] In some embodiments, the second determining unit is configured to perform at least one of the following:

[0220] Based on the first recommendation reference information of each indicator, determining an indicator to be recommended from the plurality of indicators;

[0221] Based on the first recommended reference information of each indicator, multiple candidate indicators are determined from multiple indicators, and the evaluation results corresponding to each candidate indicator are fused. Based on the fused evaluation results, a fused indicator to be recommended is generated. The fused indicator is used to comprehensively evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks from multiple evaluation dimensions.

[0222] In some embodiments, the second determining unit is configured to:

[0223] Based on the first recommended reference information of each indicator, determining a plurality of candidate indicators from the plurality of indicators;

[0224] Based on the evaluation result corresponding to each candidate indicator and the fusion weight corresponding to each candidate indicator, the evaluation result corresponding to each candidate indicator is fused, and based on the fused evaluation result, a fusion indicator to be recommended is generated.

[0225] In some embodiments, the second determining unit is configured to:

[0226] Based on the evaluation result corresponding to each candidate indicator and the multiple groups of reference fusion weights corresponding to the multiple candidate indicators, the evaluation result corresponding to each candidate indicator is fused multiple times, and based on the multiple evaluation results obtained after the multiple fusions, multiple candidate fusion indicators are generated, wherein the group of reference fusion weights corresponding to the multiple candidate indicators includes the reference fusion weight corresponding to each candidate indicator;

[0227] Based on multiple candidate fusion indicators, reference time series, and multiple annotated prediction time series, obtaining first recommended reference information of each candidate fusion indicator, wherein the first recommended reference information of the candidate fusion indicator indicates consistency between an evaluation result obtained by performing a priority evaluation on a time series prediction method corresponding to the multiple annotated prediction time series based on the candidate fusion indicator and the annotated information;

[0228] Based on the first recommendation reference information of each candidate fusion indicator, a fusion indicator to be recommended is determined from a plurality of candidate fusion indicators.

[0229] In some embodiments, the apparatus further comprises:

[0230] A third acquisition module is used to acquire at least one time series feature based on the real time series of the time series prediction task, where the at least one time series feature indicates at least one of the following: periodicity, stationarity, number of zero values, dimensional difference, and number of outliers of the time series of the event;

[0231] The second determination module is used to determine the indicator to be recommended based on at least one time series feature.

[0232] In some embodiments, the second determining module is used to:

[0233] Based on at least one time series feature and multiple indicators of the time series prediction task, determine second recommended reference information for each indicator, where the second recommended reference information of the indicator indicates the adaptation between the time series of the event and the indicator;

[0234] Based on the second recommendation reference information of each indicator, the indicator to be recommended is determined.

[0235] Through the above-mentioned device, a reference time series of a time series prediction task and multiple annotated prediction time series are obtained, and the indicators to be recommended are determined based on the obtained time series, wherein the multiple annotated prediction time series include annotation information to indicate the priority of the time series prediction method corresponding to each annotated prediction time series. Therefore, the annotation information can accurately reflect the prediction effect of the time series prediction method corresponding to each annotated prediction time series. Furthermore, the indicators determined using this annotation information can accurately evaluate the prediction effect when different time series prediction methods are used to perform time series prediction tasks, thereby improving the expected conformity of the evaluation results for the time series prediction tasks.

[0236] In addition, in the above device, the first acquisition module 1201, the second acquisition module 1202 and the first determination module 1203 can be implemented by software or by hardware. Exemplarily, the implementation of the first acquisition module 1201 is described below by taking the first acquisition module 1201 as an example. Similarly, the implementation of other modules can refer to the implementation of the first acquisition module 1201.

[0237] As an example of a software functional unit, the first acquisition module 1201 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first acquisition module 1201 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region can include multiple AZs.

[0238] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0239] As an example of a hardware functional unit, the first acquisition module 1201 may include at least one computing device. Alternatively, the first acquisition module 1201 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0240] The multiple computing devices included in the first acquisition module 1201 can be distributed in the same region or in different regions. The multiple computing devices included in the first acquisition module 1201 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first acquisition module 1201 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0241] It should be noted that: the indicator recommendation device for the time series prediction task provided in the above embodiment only uses the division of the above functional modules as an example when performing the indicator recommendation for the time series prediction task. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the indicator recommendation device for the time series prediction task provided in the above embodiment and the indicator recommendation method embodiment for the time series prediction task belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0242] In this application, the words such as "first" and "second" are used to distinguish the same or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second" and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various examples described, the first indicator can be referred to as the second indicator, and similarly, the second indicator can be referred to as the first indicator. Both the first indicator and the second indicator can be indicators, and in some cases, can be separate and different indicators.

[0243] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more, for example, a plurality of indicators means two or more indicators.

[0244] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0245] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions in the embodiments of the present application are generated.

[0246] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0247] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for recommending indicators for time series prediction tasks, characterized in that: The method comprises: Acquire a reference time series of a time series prediction task, wherein the time series prediction task indicates predicting a time series of events; Acquire multiple annotated prediction time series of the time series prediction task, wherein the reference time series and the annotated prediction time series have an overlapping time period, and the time series prediction methods corresponding to any two annotated prediction time series are different, and the multiple annotated prediction time series include annotation information, and the annotation information indicates the priority of each annotated prediction time series; Based on the reference time series and the multiple annotated prediction time series, an indicator to be recommended is determined, and the indicator is used to evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods.

2. The method according to claim 1, characterized in that The obtaining of a reference time series for a time series prediction task includes: obtaining the reference time series uploaded by a user.

3. The method according to claim 1 or 2, characterized in that: The obtaining of a plurality of annotated predicted time series of the time series prediction task includes any of the following: Obtaining the predicted time series of the multiple annotations uploaded by the user; Obtain a real time series of the time series prediction task uploaded by a user, based on the real time series, use multiple different time series prediction methods to perform the time series prediction task to obtain multiple predicted time series, and based on the multiple predicted time series, obtain the annotation information to obtain the multiple annotated prediction time series.

4. The method according to any one of claims 1 to 3, characterized in that The determining the indicator to be recommended based on the reference time series and the plurality of annotated predicted time series includes: Determining the overlapping time period of the reference time series and each annotated prediction time series; Based on the time series corresponding to the overlapping time periods, the indicator to be recommended is determined.

5. The method according to any one of claims 1 to 4, characterized in that The determining the indicator to be recommended based on the reference time series and the plurality of annotated predicted time series includes: Based on the reference time series, the multiple annotated prediction time series, and the multiple indicators of the time series prediction task, determining first recommended reference information for each indicator, wherein the first recommended reference information of the indicator indicates consistency between an evaluation result obtained by performing a priority evaluation on the time series prediction methods corresponding to the multiple annotated prediction time series based on the indicator and the annotated information; Based on the first recommendation reference information of each indicator, the indicator to be recommended is determined.

6. The method according to claim 5, characterized in that The determining, based on the reference time series, the multiple annotated predicted time series, and the multiple indicators of the time series prediction task, the first recommended reference information for each indicator includes: Based on the reference time series, the multiple annotated prediction time series, and a first indicator, obtaining an evaluation result corresponding to the first indicator, where the first indicator refers to any one of the multiple indicators; Based on the evaluation result corresponding to the first indicator and the annotation information, obtaining a recommended score value of the first indicator, where the recommended score value of the first indicator indicates consistency between the evaluation result corresponding to the first indicator and the annotation information; Based on the recommendation score value of the first indicator, first recommendation reference information of the first indicator is determined.

7. The method according to claim 5 or 6, characterized in that: The determining of the indicator to be recommended based on the first recommendation reference information of each indicator includes at least one of the following: Based on the first recommendation reference information of each indicator, determining an indicator to be recommended from the multiple indicators; Based on the first recommended reference information of each indicator, multiple candidate indicators are determined from the multiple indicators, the evaluation results corresponding to each candidate indicator are fused, and based on the fused evaluation results, a fused indicator to be recommended is generated, and the fused indicator is used to comprehensively evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods from multiple evaluation dimensions.

8. The method according to claim 7, characterized in that The step of determining a plurality of candidate indicators from the plurality of indicators based on the first recommendation reference information of each indicator, fusing the evaluation results of each candidate indicator, and generating a fusion indicator to be recommended based on the fused evaluation results includes: Based on the first recommended reference information of each indicator, determining a plurality of candidate indicators from the plurality of indicators; Based on the evaluation result corresponding to each candidate indicator and the fusion weight corresponding to each candidate indicator, the evaluation result corresponding to each candidate indicator is fused, and based on the fused evaluation result, a fusion indicator to be recommended is generated.

9. The method according to claim 8, characterized in that The step of fusing the evaluation results corresponding to each candidate indicator based on the evaluation results corresponding to each candidate indicator and the fusion weight corresponding to each candidate indicator, and generating a fusion indicator to be recommended based on the fused evaluation results, includes: Based on the evaluation result corresponding to each candidate indicator and the multiple groups of reference fusion weights corresponding to the multiple candidate indicators, the evaluation result corresponding to each candidate indicator is fused multiple times, and based on the multiple evaluation results obtained after the multiple fusions, multiple candidate fusion indicators are generated, wherein the group of reference fusion weights corresponding to the multiple candidate indicators includes the reference fusion weight corresponding to each candidate indicator; Based on the multiple candidate fusion indicators, the reference time series, and the multiple annotated prediction time series, obtaining first recommended reference information for each candidate fusion indicator, wherein the first recommended reference information of the candidate fusion indicator indicates consistency between an evaluation result obtained by performing a priority evaluation on the time series prediction methods corresponding to the multiple annotated prediction time series based on the candidate fusion indicator and the annotated information; Based on the first recommendation reference information of each candidate fusion indicator, a fusion indicator to be recommended is determined from the multiple candidate fusion indicators.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Based on the real time series of the time series prediction task, obtaining at least one time series feature, wherein the at least one time series feature indicates at least one of the following: periodicity, stationarity, number of zero values, dimensional difference, and number of outliers of the time series of the event; Based on the at least one time series feature, an indicator to be recommended is determined.

11. The method according to claim 10, characterized in that The determining the indicator to be recommended based on the at least one time series feature includes: Based on the at least one time series feature and the multiple indicators of the time series prediction task, determine second recommended reference information for each indicator, wherein the second recommended reference information of the indicator indicates the adaptation between the time series of the event and the indicator; Based on the second recommendation reference information of each indicator, the indicator to be recommended is determined.

12. An indicator recommendation device for a time series prediction task, characterized in that: The device comprises: A first acquisition module is used to acquire a reference time series of a time series prediction task, wherein the time series prediction task indicates to predict a time series of an event; A second acquisition module is used to acquire a plurality of annotated prediction time series of the time series prediction task, wherein the reference time series and the annotated prediction time series have an overlapping time period, and the time series prediction methods corresponding to any two annotated prediction time series are different, and the plurality of annotated prediction time series include annotation information, and the annotation information indicates the priority of the time series prediction method corresponding to each annotated prediction time series; The first determination module is used to determine the index to be recommended based on the reference time series and the multiple annotated prediction time series, and the index is used to evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods.

13. The device according to claim 12, characterized in that The first acquisition module is used to: acquire the reference time series uploaded by the user.

14. The device according to claim 12 or 13, characterized in that The second acquisition module is used for any of the following: Obtaining the predicted time series of the multiple annotations uploaded by the user; Obtain a real time series of the time series prediction task uploaded by a user, based on the real time series, use multiple different time series prediction methods to perform the time series prediction task to obtain multiple predicted time series, and based on the multiple predicted time series, obtain the annotation information to obtain the multiple annotated prediction time series.

15. The device according to any one of claims 12 to 14, characterized in that The first determining module is used to: Determining the overlapping time period of the reference time series and each annotated prediction time series; Based on the time series corresponding to the overlapping time periods, the indicator to be recommended is determined.

16. The device according to any one of claims 12 to 15, characterized in that The first determining module includes: A first determining unit is used to determine first recommended reference information for each indicator based on the reference time series, the multiple annotated prediction time series, and the multiple indicators of the time series prediction task, wherein the first recommended reference information of the indicator indicates consistency between an evaluation result obtained by performing a priority evaluation on the time series prediction methods corresponding to the multiple annotated prediction time series based on the indicator and the annotated information; The second determining unit is used to determine the indicator to be recommended based on the first recommendation reference information of each indicator.

17. The device according to claim 16, characterized in that The first determining unit is configured to: Based on the reference time series, the multiple annotated prediction time series, and a first indicator, obtaining an evaluation result corresponding to the first indicator, where the first indicator refers to any one of the multiple indicators; Based on the evaluation result corresponding to the first indicator and the annotation information, obtaining a recommended score value of the first indicator, where the recommended score value of the first indicator indicates consistency between the evaluation result corresponding to the first indicator and the annotation information; Based on the recommendation score value of the first indicator, first recommendation reference information of the first indicator is determined.

18. The device according to claim 16 or 17, characterized in that The second determining unit is configured to: Based on the first recommendation reference information of each indicator, determining an indicator to be recommended from the multiple indicators; Based on the first recommended reference information of each indicator, multiple candidate indicators are determined from the multiple indicators, the evaluation results corresponding to each candidate indicator are fused, and based on the fused evaluation results, a fused indicator to be recommended is generated, and the fused indicator is used to comprehensively evaluate the prediction effect when the time series prediction task is performed using different time series prediction methods from multiple evaluation dimensions.

19. The device according to claim 18, characterized in that The second determining unit is configured to: Based on the first recommended reference information of each indicator, determining a plurality of candidate indicators from the plurality of indicators; Based on the evaluation result corresponding to each candidate indicator and the fusion weight corresponding to each candidate indicator, the evaluation result corresponding to each candidate indicator is fused, and based on the fused evaluation result, a fusion indicator to be recommended is generated.

20. The device according to claim 19, characterized in that The second determining unit is configured to: Based on the evaluation result corresponding to each candidate indicator and the multiple groups of reference fusion weights corresponding to the multiple candidate indicators, the evaluation result corresponding to each candidate indicator is fused multiple times, and based on the multiple evaluation results obtained after the multiple fusions, multiple candidate fusion indicators are generated, wherein the group of reference fusion weights corresponding to the multiple candidate indicators includes the reference fusion weight corresponding to each candidate indicator; Based on the multiple candidate fusion indicators, the reference time series, and the multiple annotated prediction time series, obtaining first recommended reference information for each candidate fusion indicator, wherein the first recommended reference information of the candidate fusion indicator indicates consistency between an evaluation result obtained by performing a priority evaluation on the time series prediction methods corresponding to the multiple annotated prediction time series based on the candidate fusion indicator and the annotated information; Based on the first recommendation reference information of each candidate fusion indicator, a fusion indicator to be recommended is determined from the multiple candidate fusion indicators.

21. The device according to any one of claims 12 to 20, characterized in that The device also includes: A third acquisition module is used to acquire at least one time series feature based on the real time series of the time series prediction task, wherein the at least one time series feature indicates at least one of the following: periodicity, stationarity, number of zero values, dimensional difference, and number of outliers of the time series of the event; The second determination module is used to determine the indicator to be recommended based on the at least one time series feature.

22. The device according to claim 21, characterized in that The second determining module is used to: Based on the at least one time series feature and the multiple indicators of the time series prediction task, determine second recommended reference information for each indicator, wherein the second recommended reference information of the indicator indicates the adaptation between the time series of the event and the indicator; Based on the second recommendation reference information of each indicator, the indicator to be recommended is determined.

23. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the indicator recommendation method for the time series prediction task as described in any one of claims 1 to 11.

24. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing device cluster, the computing device cluster executes the indicator recommendation method for the timing prediction task as described in any one of claims 1 to 11.

25. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the indicator recommendation method for the time series prediction task as described in any one of claims 1 to 11.

Images