A method for extending a time series database analysis capability

Abstract

The application provides a method for extending time series database analysis capability, and relates to the field of data analysis, which comprises the following steps: constructing a time series data analysis system for providing time series data analysis service independently of a time series database system, and extending, upgrading and maintaining the time series data analysis system without affecting the normal operation of the time series database system; adding an adaptive interface in the time series data analysis system, and calling different time series data analysis models; establishing an analysis operator for the time series data analysis service in the time series database system, and the analysis operator calling the corresponding time series data analysis model through the adaptive interface to complete time series data analysis in the execution of an SQL query process, so that interaction between the time series database system and the time series data analysis system which are isolated from each other in the processing flow is realized, and the complexity of the analysis and processing process is shielded.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data analysis, in particular to a method for extending the analysis capability of a time series database. BACKGROUND

[0002] The analysis capability of a time series database includes simple data filtering, extraction, aggregation, and other capabilities, as well as complex data prediction, anomaly analysis, and other capabilities.

[0003] In the prediction analysis and anomaly detection of time series data (referred to as time series data), researchers have developed numerous advanced analysis algorithms with different technical characteristics and suitable for different application scenarios, such as time series data prediction, time series data anomaly detection, and time series data change point detection. These analysis algorithms usually exist in the form of data analysis toolkits in high-level programming languages such as Python and R.

[0004] Currently, data analysis toolkits embedded in time series database systems can be used in a cross-language call manner, and the analysis algorithms embedded in the time series database system can be called using SQL statements. However, this direct calling method has the following problems:

[0005] 1. Poor scalability and difficult to upgrade analysis algorithms. The extension and upgrade of analysis algorithms are independent of the time series database system. After the analysis algorithms are packaged into the time series database system, the analysis algorithms cannot be updated individually. Any extension and upgrade of analysis algorithms requires re-packaging and compiling the entire time series database software.

[0006] 2. The calling process of the analysis algorithm invades the query execution logic of the time series database system, specifically: the calling API of different analysis algorithms is very different, and the calling and logic processing process needs to be designed separately for each algorithm. Especially for pre-trained analysis algorithms and non-training analysis algorithms, differential processing is required, and the complexity of the processing logic cannot be shielded. SUMMARY

[0007] The present application provides a method for extending the analysis capability of a time series database, which solves the problems of poor scalability and difficulty in upgrading analysis algorithms, and the calling process of the analysis algorithm invading the query execution logic of the time series database system.

[0008] The application provides a method for extending the analysis capability of a time series database, which comprises: constructing a time series data analysis system running independently of a time series database system to provide time series data analysis services, so as to extend, upgrade and maintain the time series data analysis system without affecting the normal operation of the time series database system; adding an adaptation interface capable of calling different time series data analysis models in the time series data analysis system; establishing an analysis operator for the time series data analysis services in the time series database system, which complies with the operator rules of the time series database system, so as to realize interaction between the time series database system and the time series data analysis system in which the processing flows are isolated; and the analysis operator calls a corresponding time series data analysis model through the adaptation interface in the time series data analysis system to complete time series data analysis in the execution of an SQL query flow.

[0009] Preferably, the method for constructing a time series data analysis system running independently of a time series database system to provide time series data analysis services comprises: separating time series data analysis functions from an existing time series database system to form an independent time series data analysis system, wherein the time series data analysis functions are realized by time series data analysis models running in corresponding running environments.

[0010] Preferably, the method for extending, upgrading and maintaining the time series analysis system comprises: extending a new time series data analysis model for the time series data analysis system, specifically: constructing a class file of the new time series data analysis model, wherein the class file of the new time series data analysis model contains an analysis method of the new time series data analysis model; saving the class file of the new time series data analysis model under a specified directory of the time series data analysis system; loading the class file of the new time series data analysis model under the specified directory to load the new time series data analysis model into the time series data analysis system, so as to extend the time series data analysis system without affecting the normal operation of the time series database system.

[0011] Preferably, the method for adding an adaptation interface capable of calling different time series data analysis models in the time series data analysis system comprises: constructing an encapsulation class for each time series data analysis model in the time series data analysis system, and the encapsulation classes of time series data analysis models of the same type inherit from the same base class, so as to obtain an adaptation interface for adapting different time series data analysis models.

[0012] Preferably, the time series data analysis service provided by the time series database system for the time series data analysis system establishes an analysis operator complying with the time series database system operator rule to realize interaction between the time series database system and the time series data analysis system isolated in processing flow, which comprises: establishing one analysis operator for each type of time series data analysis service in the time series database system according to the time series database system operator rule, and embedding each established analysis operator into the SQL query flow in the time series database system; wherein the function of the analysis operator comprises: converting the time series data and the calling information of the time series data analysis model received from the upstream operator of the time series database system into a JSON format analysis request and sending it to the time series data analysis system, and converting the received JSON format analysis result from the time series data analysis system into an analysis result conforming to the internal data format of the time series database system and directly delivering or delivering after aggregation to the downstream operator of the time series database system.

[0013] Preferably, when the type of the time series data analysis service is prediction analysis, the analysis operator is a prediction analysis operator, and correspondingly, the time series database system in the execution of the SQL query flow, the analysis operator completes the time series data analysis by calling the corresponding time series data analysis model in the time series data analysis system through the adaptation interface, which comprises: when the user calls a time series data analysis model through a SQL statement to analyze time series data, the time series database system executes the SQL query flow; in the execution of the SQL query flow, the prediction analysis operator receives the time series data output by the upstream operator, and verifies the calling information of the time series data analysis model specified in the SQL statement; the prediction analysis operator converts the time series data and the calling information into a JSON format analysis request and sends it to the time series data analysis system, so that the time series data analysis system receives and parses the JSON format analysis request, calls the time series data analysis model indicated by the JSON format analysis request through the adaptation interface to perform prediction analysis, and returns the JSON format analysis result of the time series data analysis model to the time series database system; the prediction analysis operator receives the JSON format analysis result returned by the time series data analysis system, and converts the JSON format analysis result into an analysis result in the internal data format of the time series database system, and delivers it to the downstream operator.

[0014] Preferably, when the type of the time series data analysis service is anomaly detection, the analysis operator of the time series data analysis system is an anomaly detection operator, and accordingly, in the execution of the SQL query process, the analysis operator calls a corresponding time series data analysis model through the adaptation interface in the time series data analysis system to complete time series data analysis, including: when a user calls a time series data analysis model through a SQL statement to analyze time series data, the time series database system executes the SQL query process; in the execution of the SQL query process, the anomaly detection operator receives time series data output by an upstream operator, and checks the calling information of the time series data analysis model specified in the SQL statement; the anomaly detection operator converts the time series data and the calling information into a JSON format analysis request and sends it to the time series data analysis system, so that the time series data analysis system receives and parses the JSON format analysis request, calls the time series data analysis model indicated by the JSON format analysis request through the adaptation interface to perform anomaly detection, and returns the JSON format analysis result of the time series data analysis model to the time series database system; the anomaly detection operator converts the received JSON format analysis result into an analysis result in the internal data format of the time series database system, and according to the aggregation function specified in the SQL statement, aggregates the analysis result in the internal data format of the time series database system, and passes the aggregation result to the downstream operator.

[0015] Preferably, before the analysis operator calls a corresponding time series data analysis model through the adaptation interface in the time series data analysis system to complete time series data analysis in the execution of the SQL query process by the time series database system, the method further comprises: adding address information of the time series data analysis system to a metadata node of the time series database system; and the time series database system acquires and saves metadata information related to the time series data analysis model provided by the time series data analysis system from the time series data analysis system according to the address information of the time series data analysis system added to the metadata node.

[0016] Preferably, the time series database system is deployed on a server with a CPU, and the time series data analysis system is deployed on a server with a GPU.

[0017] Preferably, the relationship between the time series database system and the time series data analysis system is one of one-to-one, one-to-many, many-to-one, and many-to-many.

[0018] The timing database analysis capability expansion method provided by the application, by constructing a timing data analysis system independent of the timing database system, makes the expansion, upgrade and maintenance of the timing data analysis system not affect the normal operation of the timing database system, the timing data analysis model (i.e. analysis algorithm) in the timing data analysis system has good expansibility, high flexibility and low upgrade difficulty; in addition, by setting an adaptive interface in the timing data analysis system, the calling of different timing data analysis models in the timing data analysis system is realized, and the calling logic and use interface do not need to be developed separately for each timing analysis model; in addition, by establishing an analysis operator in the timing database system for the timing data analysis service, the SQL query process of the timing database system and the complex processing logic in the timing data analysis system are isolated from each other. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the timing database analysis capability expansion method provided by the application;

[0020] Figure 2 is an architecture diagram of the timing data analysis system provided by the application;

[0021] Figure 3 is a schematic diagram of a JSON format analysis request for performing prediction analysis provided by the application;

[0022] Figure 4 is a response Figure 3 is a schematic diagram of a JSON format analysis result of the analysis request;

[0023] Figure 5 is a relationship diagram between the timing database system and the timing data analysis system provided by the application;

[0024] Figure 6 is a processing flowchart of the analysis call provided by the application. DETAILED DESCRIPTION

[0025] The embodiments of the application will be described in detail below with reference to the accompanying drawings, and it should be understood that the embodiments described below are only used to illustrate and explain the application, and are not used to limit the application.

[0026] The application provides a timing database analysis capability expansion method for expanding the analysis capability of the timing database system, including but not limited to timing data prediction analysis, timing data anomaly detection and other complex timing data analysis capabilities.

[0027] Referring to Figure 1 , the timing database analysis capability expansion method provided by the application includes the following steps:

[0028] Step S101: constructing a time series data analysis system for providing time series data analysis services independently of a time series database system to extend, upgrade and maintain the time series data analysis system without affecting normal operation of the time series database system.

[0029] In one aspect, complex time series data analysis functions, such as prediction analysis functions and anomaly detection functions, can be separated from an existing time series database system to form an independent time series data analysis system, wherein the time series data analysis functions are implemented by time series data analysis models running in corresponding operating environments. In another aspect, new time series data analysis models (i.e., new analysis algorithms) providing time series data analysis functions can be extended to the independent time series data analysis system, specifically: constructing a class file with a name based on the name of the new time series data analysis model for the new time series data analysis model, wherein the class file of the new time series data analysis model contains analysis methods of the new time series data analysis model, then saving the class file of the new time series data analysis model under a specified directory of the time series data analysis system, and loading the class file of the new time series data analysis model under the specified directory to load the new time series data analysis model into the time series data analysis system, thereby extending the analysis capabilities of the time series data analysis system without affecting normal operation of the time series database system.

[0030] In the present application, since the time series data analysis system and the time series database system are independent of each other, the upgrade and maintenance of the time series data analysis models in the time series data analysis system are not limited by the time series database system, so the time series data analysis system has good scalability, high flexibility and easier upgrade and maintenance.

[0031] Step S102: adding an adaptation interface capable of calling different time series data analysis models in the time series data analysis system.

[0032] Specifically, an encapsulation class is constructed for each time series data analysis model in the time series data analysis system, and the encapsulation classes of the same type of time series data analysis models inherit from the same base class to obtain an adaptation interface that adapts to different time series data analysis models.

[0033] In addition, the time series data analysis system further comprises a request processing and adaptation library, and functions of the request processing and adaptation library include: (1) receiving and parsing an analysis request from the time series database system; (2) according to calling information of a time series data analysis model configured in the analysis request, such as a model name and calling parameters, calling the time series data analysis model indicated in the analysis request in different time series data analysis models of the time series data analysis system through an adaptation interface to analyze time series data in the analysis request; and (3) returning analysis results obtained by the time series data analysis model from analyzing the time series data to the time series database system.

[0034] The request processing and adaptation library and the adaptation interface integrate different time series data analysis models into standardized analysis services, and improve the support capability of the time series data analysis system for different time series data analysis models, so that it is not necessary to separately develop calling logic and use interfaces for each time series analysis model.

[0035] Step S103: establishing an analysis operator in the time series database system for the time series data analysis service, the analysis operator complying with an operator rule of the time series database system, so as to realize interaction between the time series database system and the time series data analysis system which are isolated in processing flows.

[0036] According to the operator rule of the time series database system, an analysis operator is established in the time series database system for each type of time series data analysis service, and each established analysis operator is embedded in a SQL query flow in the database system, so that when a user specifies a time series data analysis service in a SQL statement, a SQL plan generator of the time series database system adds a corresponding analysis operator in a syntax tree. The operator rule adopts an existing technology, and will not be described herein. Functions of the analysis operator include: converting time series data and calling information of a time series data analysis model received from an upstream operator of the time series database system into a JSON format analysis request and sending the analysis request to the time series data analysis system, and converting a JSON format analysis result received from the time series data analysis system into an analysis result conforming to an internal data format of the time series database system and directly delivering or delivering after aggregation processing to a downstream operator of the time series database system. Obviously, integrating the analysis operator with the above functions into the SQL query flow of the time series database system can not only shield the query processing flow of the time series database system and the complex analysis processing logic of the time series data analysis system, but also utilize the analysis capability of the time series data analysis system to let the time series database system obtain processing results of different analysis tasks.

[0037] Step S104: The time series database system executes a SQL query process, and the analysis operator invokes a corresponding time series data analysis model through the adaptation interface in the time series data analysis system to complete time series data analysis.

[0038] For example, when the type of the time series data analysis service is prediction analysis, the analysis operator is a prediction analysis operator. Accordingly, the step S104 includes: when a user invokes a time series data analysis model through a SQL statement to analyze time series data, the time series database system executes a SQL query process; in the SQL query process, the prediction analysis operator receives time series data output by an upstream operator, and checks invocation information of the time series data analysis model specified in the SQL statement; the prediction analysis operator converts the time series data and the invocation information into a JSON format analysis request and sends it to the time series data analysis system, so that the time series data analysis system receives and parses the JSON format analysis request through its request processing and adaptation library, invokes the time series data analysis model indicated by the JSON format analysis request through the adaptation interface to perform prediction analysis, and returns a JSON format analysis result of the time series data analysis model to the time series database system; the prediction analysis operator receives the JSON format analysis result returned by the time series data analysis system, converts the JSON format analysis result into an analysis result in the internal data format of the time series database system, and delivers it to a downstream operator.

[0039] For example, when the type of the time series data analysis service is anomaly detection, the analysis operator of the time series data analysis system is an anomaly detection operator, and accordingly, the step S104 includes: when a user calls a time series data analysis model to analyze time series data through a SQL statement, the time series database system executes a SQL query process; in the execution of the SQL query process, the anomaly detection operator receives time series data output by an upstream operator, and checks calling information of the time series data analysis model specified in the SQL statement; the anomaly detection operator converts the time series data and the calling information into a JSON format analysis request and sends it to the time series data analysis system, so that the time series data analysis system receives and parses the JSON format analysis request by using its request processing and adaptation library, calls the time series data analysis model indicated by the JSON format analysis request through the adaptation interface to perform anomaly detection, returns the JSON format analysis result of the time series data analysis model to the time series database system; the anomaly detection operator converts the received JSON format analysis result into an analysis result in the internal data format of the time series database system, and performs aggregation processing on the analysis result in the internal data format of the time series database system according to the aggregation function specified in the SQL statement, and passes the aggregation result to the downstream operator.

[0040] The present application is not limited to the prediction analysis operator and the anomaly detection operator, and according to actual analysis needs, analysis operators of other types of time series data analysis services can also be established, and corresponding time series data analysis models in the time series data analysis system can be called to provide time series data analysis services.

[0041] Further, in order to enable the time series database system to know that the time series data analysis system exists and to dynamically call time series data analysis models in the time series data analysis system in the SQL query process, the independently deployed and run time series data analysis system needs to be dynamically registered in the time series database system. Specifically, before the analysis operator calls corresponding time series data analysis models through the adaptation interface in the time series data analysis system to complete time series data analysis in the execution of the SQL query process by the time series database system, the address information of the time series data analysis system is added to the metadata node of the time series database system, and the time series database system acquires and saves metadata information related to time series data analysis models provided by the time series data analysis system from the time series data analysis system according to the address information of the time series data analysis system added to the metadata node.

[0042] Further, in the prior art scheme of embedding the analysis algorithm into the time series database system, since the analysis algorithm inference analysis process is a computationally intensive task, requiring a large amount of CPU computing power, even relying on GPU acceleration, frequent calling of such analysis algorithms will bring a large computing load pressure to the server of the time series database system, and further affect the normal operation of other services. Therefore, the time series database system of the present application is deployed on a server with CPU, ensuring that the server handles regular database functions. The time series data analysis system is deployed on a server with GPU, and the performance of the time series data analysis service is improved through GPU.

[0043] Further, the relationship between the time series database system and the time series data analysis system is one of one-to-one, one-to-many, many-to-one, and many-to-many. One time series data analysis system can provide one or more types of time series data analysis services to one or more time series database systems through cooperation with one or more time series database systems. One time series database system can call the time series data analysis model of one or more time series data analysis systems by registering one or more time series data analysis systems. In the present application, there is no information interaction between multiple time series data analysis systems, and each time series data analysis system provides stateless time series data analysis services, so the time series data analysis system can be quickly expanded horizontally.

[0044] The following will be described in detail from the aspects of the architecture of the time series data analysis system, the message exchange format between the time series database system and the time series data analysis system, the interaction between the time series database system and the time series data analysis system, the processing flow of the analysis call, and the way of adding new time series data analysis models to the time series data analysis system. Figures 2 to 6

[0045] I. Architecture of the time series data analysis system

[0046] Referring to Figure 2 The time series data analysis system that can be independently deployed and run includes:

[0047] 1. Time series data analysis model

[0048] The time series data analysis model can provide multiple types of analysis services, and different time series data analysis services can be provided for different scenarios, including prediction analysis, anomaly detection, missing data completion, pattern classification, and other types of analysis service functions. It is the carrier of analysis services.

[0049] The time series data analysis model includes machine learning models, time series base models, and deep learning models according to the model type. Among them, the time series base model is a large-scale pre-training model designed for time series data, which realizes multi-task generalization ability through a general architecture.

[0050] ​2. Base runtime library

[0051] The base runtime library provides the basic environment for the running of time series data analysis models, usually a variety of types of Python runtime libraries, to drive the running of time series data analysis models including machine learning models, time series base models, and deep learning models.

[0052] 3. Request processing and adaptation library

[0053] The request processing and adaptation library is responsible for receiving and parsing JSON format request messages, and converting the analysis results after analysis into JSON format and returning to the requestor.

[0054] In order to shield the differences of different time series data analysis models and achieve unified calling, the time series data analysis system of the present application realizes unified management and calling of different analysis models through the class encapsulation and inheritance mechanism of Python.

[0055] For each time series data analysis model, a corresponding Python class is established to encapsulate the time series data analysis model, and the class name is usually based on the model name. For example, the encapsulation class corresponding to the prediction analysis model Holtwinters is _HoltWintersService, and the encapsulation class corresponding to the prediction analysis model ARIMA is _ArimaService.

[0056] The same type of time series data analysis model inherits the corresponding base class, such as the Holtwinters, ARIMA, etc. prediction analysis class inherits from the AbstractForecastService class, and the anomaly detection class inherits from the AbstractAnomalyDetectionService class, so that the same type of model can have a unified basic structure.

[0057] Based on the requirements of the base class for the subclass, all encapsulation classes for the model need to implement two key functions: 1. The set_param() function sets the necessary parameters for the model running, and 2. The execute() function executes the model analysis process. Therefore, it can be guaranteed that an instantiated model object that follows a unified interface is obtained in the model calling process.

[0058] The time series database system specifies the analysis model to be used through a JSON format request message. The request processing and adaptation library of the time series analysis system parses the JSON format request message to obtain the model name to be called, for example: specify Holtwinters as the prediction analysis model, then instantiate the encapsulation class _HoltWintersService corresponding to Holtwinters, generate a _HoltWintersService object, then call the set_param() function of the object to pass in the calling parameters, and then call the execute() function of the object. The analysis model executes the analysis process in the running environment.

[0059] Through the analysis model encapsulation, class inheritance system, and adaptation interfaces set_param() and execute(), regardless of whether the analysis model is Holtwinters, ARIMA, or other models, external calls to the analysis model do not need to care about model differences, which realizes shielding of differences between different analysis models for external time series database systems and unified management of different analysis models internally.

[0060] II. Message exchange format between the time series database system and the time series data analysis system

[0061] The time series data analysis system provides time series data prediction, time series data anomaly detection, and other analysis functions. It provides a unified service request interface to call different types of time series data analysis models, such as traditional machine learning analysis models, pre-trained deep learning models, and the most advanced time series base models.

[0062] The JSON format analysis request contains the time series data to be analyzed and the calling information of the time series data analysis model (such as model name, calling parameters, etc.). The analysis request is sent to the time series data analysis system through HTTP. Taking a prediction analysis request as an example, see Figure 3 The "algo" field specifies the model used for prediction analysis. Obviously, by modifying the content of the "algo" field, different model names can be set, that is, different time series data analysis models in the time series data analysis system can be called. The "data" field is the time series data for prediction analysis, and other fields are option information of the time series data analysis model. The analysis result is also returned to the requesting time series database system in JSON format. A typical prediction analysis return result is shown in Figure 4 .

[0063] The JSON format is a standard format and is mainly used for information exchange between different systems. Therefore, the application uses the JSON format for data exchange and information transmission between the time series database system and the time series data analysis system. It should be noted that this format is not used within the system because the expression efficiency is too low, the additional overhead is high, and the system performance is affected.

[0064] III. Interaction between the time series database system and the time series data analysis system

[0065] 1. The time series data analysis system is registered to the time series database system so that the time series database system knows that the time series data analysis system exists and sends analysis requests to the time series data analysis system. The registration process is as follows:

[0066] Step 1.1, the system administrator uses a command to add the IP:PORT information of the time series data analysis system service to the time series database system metadata node.

[0067] Step 1.2, after the addition is completed, the time series database system actively requests the time series data analysis system to obtain the metadata information of the time series data analysis model provided by it, such as model basic attributes (such as name, version, creation time), input and output data characteristics (such as time series field, time granularity, data type), architecture parameters (such as network layer number, neuron number, activation function), training configuration (such as optimizer, loss function, iteration number), evaluation index (such as MAE, RMSE, MAPE) and the like.

[0068] Step 1.3, after the time series database system obtains the metadata information of the time series data analysis model, it caches it in the time series database system metadata node and synchronizes it to other nodes of the time series database system, such as computing nodes, storage nodes, backup nodes, etc.

[0069] 2. The relationship between the time series data analysis system and the time series database system can be many-to-many, as shown in Figure 5 That is, one time series database system registers multiple time series data analysis systems as needed. One time series data analysis system can also be registered to multiple time series database systems.

[0070] The time series database system is deployed on a server with a high-performance CPU. The analysis system is deployed on a server equipped with a high-performance GPU, which uses GPU acceleration to analyze the machine learning / deep learning process. Among them, CPU and GPU use the existing high-performance CPU and GPU. Registering the time series data analysis system in multiple time series database systems can share high-performance GPU servers among multiple time series database systems, fully utilizing the performance advantages of hardware.

[0071] 3. For each type of time series data analysis model, a corresponding operator needs to be established within the time series database system and integrated into the SQL query process within the time series database system.

[0072] In a database system, an operator is a basic functional unit for operating data when executing a SQL query, usually represented as a specific operation step in a query plan. For example, a scan operator that is responsible for scanning all table data, an aggregate operator that is responsible for aggregating data to generate results, a project operator that is responsible for extracting table data by column, and the like.

[0073] In order to support time series data prediction analysis and time series data anomaly detection, the present application adds a forecast operator for processing prediction analysis logic and an anomaly detection operator for processing anomaly detection logic in the time series database system.

[0074] The main functions of the forecast operator are as follows:

[0075] (1) Receive time series data read by an upstream operator (usually a scan operator, etc.);

[0076] (2) Verify the call parameter information (or called information) of the time series data analysis model specified in the SQL statement;

[0077] (3) Convert the time series data and the call information into a request message in the format shown in the JSON format data, and send it to the time series data analysis system. Figure 3

[0078] (4) After the forecast operator receives the analysis result in JSON format, it converts it into an internal format used by the time series database system (used during calculation) and passes it to the downstream operator to continue the query execution process.

[0079] The anomaly detection operator is an aggregate operator that can aggregate time series data within an anomaly (data) point window formed by continuous anomaly data points according to a user-specified aggregation function, and its main functions are as follows:

[0080] (1) Receive time series data read by an upstream operator (usually a scan operator, etc.);

[0081] (2) Verify the call parameter information (or called information) of the time series data analysis model provided in the SQL statement; ​

[0082] (3) The time series data and the calling information are converted into a JSON format analysis request, and the analysis request is sent to the time series data analysis system;

[0083] (4) After the anomaly detection operator receives the analysis result in the JSON format, the analysis result is converted into an internal format of the time series database system, and an aggregated result is generated using a user-specified aggregation function;

[0084] (5) The generated result is passed to a downstream operator.

[0085] IV. Processing flow of an analysis call

[0086] 1. Referring to Figure 6 , a main execution flow of the prediction analysis operator is as follows:

[0087] (1) Data input of a downstream operator is received.

[0088] Data used for analysis is usually sourced from an upstream operator, and the upstream operator of the prediction analysis operator is usually a scan operator, which is responsible for scanning data in a table and passing the data to the prediction analysis operator for time series data prediction analysis.

[0089] (2) The prediction analysis operator is responsible for receiving time series data obtained by the scan operator, and performing legality check on the received time series data and calling information used for calling a time series data analysis model. After the check passes, the time series data used for analysis and the calling information are converted into a JSON format analysis request, which is sent to the time series data analysis system. Then, the prediction analysis operator itself is in a blocking state to wait for a message returned by the time series data analysis system.

[0090] (3) After the time series data analysis system receives the analysis request of the time series data, the analysis request is parsed, and a time series data analysis model is adapted according to the calling information for analysis processing. After the analysis is completed, a JSON format analysis result is returned.

[0091] (4) The prediction analysis operator receives the JSON format analysis result returned by the time series data analysis system, converts the analysis result into a data structure in an internal format of the time series database system. The blocking state of the prediction analysis operator is released, and then the analysis result is continuously passed to a downstream operator of the prediction analysis operator, i.e., a projection operator, to continue the query processing flow.

[0092] 2. Similar to the main execution flow of the prediction analysis operator, a main execution flow of the anomaly detection operator is as follows:

[0093] (1) Data input of an upstream operator is received.

[0094] The data used for analysis is usually sourced from an upstream operator. The upstream operator of the anomaly detection operator is usually a scan operator, which is responsible for scanning the data in the table and passing it to the anomaly detection operator for time series data anomaly detection.

[0095] (2) The anomaly detection operator is responsible for receiving time series data obtained by the scan operator, and performing legality checks on the received time series data and invocation information for invoking the time series data analysis model. After the checks pass, the anomaly detection operator converts the analysis used time series data and invocation information into a JSON format analysis request and sends it to the time series data analysis system. After that, the anomaly detection operator itself is in a blocked state to wait for the message returned by the time series data analysis system.

[0096] (3) After the time series data analysis system receives the analysis request for the time series data, it parses the analysis request and adapts the time series data analysis model for analysis processing according to the invocation information. After the analysis is complete, the time series data analysis system returns a JSON format analysis result.

[0097] (4) The anomaly detection operator receives the JSON format analysis result returned by the time series data analysis system and converts it into a data structure inside the time series database system. The anomaly detection operator is then unblocked, and the analysis result is aggregated and passed to the downstream operator of the anomaly detection operator, i.e., the projection operator, to continue the query processing flow.

[0098] Five. Ways to add new time series data analysis models to the time series data analysis system

[0099] The steps to extend the time series data analysis model are as follows:

[0100] Step 1: Add a class starting with an underscore (_) in the time series data analysis system. The class needs to inherit from the AbstractForecastService class and implement the virtual methods defined in it.

[0101] Step 2: Set appropriate name (the name is the invocation name of the model) and description information for the new class.

[0102] Step 3: Implement the execute method for the added class. The execute method is the main method of the time series data analysis model. The specific logic implementation of the time series data analysis model needs to be executed in this method. At the end of the execute method, the time series data prediction analysis result needs to be returned in the format of <timestamp column><prediction data column>.

[0103] The added time series data analysis model class file is saved in a directory designated by the time series data analysis system, for example, in the. / lib / taosanalytics / algo / fc / directory.

[0104] The time series data analysis model in the directory is loaded by using a command or restarting the time series data analysis system, so that the newly added time series data analysis model is loaded into the time series data analysis system and can provide time series data analysis services externally.

[0105] Since the time series data analysis system is dynamically registered in the time series database system, the time series database system can obtain the metadata information of the newly added time series data analysis model. When the user needs to call the newly added time series data analysis model, the "algo" field in the JSON format analysis request is modified to the newly added time series data analysis model, so that the newly added time series data analysis model can be called.

[0106] It should be noted that the time series data analysis system does not save the information related to the query process, does not record the user information, provides stateless time series data analysis services, and there is no information interaction between different analysis systems, so the time series data analysis system can be quickly horizontally expanded.

[0107] The present application has the following technical effects:

[0108] 1. The time series database system as the core of the application system is usually not allowed to restart or shut down in the production environment, while the time series data analysis system needs to be retrained, deployed, upgraded, and the like according to the requirements of the application. The present application separates the time series data analysis function from the time series database system to form an independent time series data analysis system, so that the expansion, upgrade, maintenance, and operation and maintenance of the time series data analysis system do not affect the normal operation of the core system (time series database system).

[0109] 2. The present application separates the complexity of time series data analysis from the time series database system, and the prediction analysis operator and the anomaly detection operator are only responsible for information interaction with the time series data analysis system and do not involve analysis and processing process, and the prediction analysis operator and the anomaly detection operator comply with the rules of the time series database operator, so they can be embedded in the SQL query execution plan of the time series database system. By isolating the complexity of the analysis and processing process through the prediction analysis operator and the anomaly detection operator, the influence of the time series data analysis model adjustment on the query processing process of the time series database system can be avoided.

[0110] 3. The time series data analysis system of the present application can be deployed on a server with GPU, and the use of GPU can greatly improve the performance of time series data analysis services. The time series database system is deployed on a CPU server, which can ensure high-performance processing of time series data analysis functions and conventional database functions.

[0111] 4. The time series data analysis system of the present application can be coordinated with one or more time series database systems to provide various types of time series data analysis services.

[0112] 5. The time series data analysis system of the present application can receive and parse standardized JSON format request messages, and call the time series data analysis model indicated by the JSON format request message for data analysis through an adaptive interface, without developing calling logic and using interfaces for each type of analysis model.

[0113] 6. After adding a new time series data analysis model, the user can directly call the new time series data analysis model through an SQL statement, without the need to upgrade / stop / redeploy the time series database system.

[0114] Although the present application has been described in detail above, the present application is not limited thereto, and various modifications can be made by those skilled in the art according to the principles of the present application. Therefore, any modifications made in accordance with the principles of the present application should be understood to fall within the scope of the present application.

Claims

1. A method for scaling temporal database analytics capabilities, the method comprising: The method comprises: building a time series data analysis system for providing stateless time series data analysis services independently of a time series database system, so as to extend, upgrade and maintain the time series data analysis system without affecting normal operation of the time series database system; adding an adaptation interface capable of calling different time series data analysis models in the time series data analysis system; establishing an analysis operator for each type of stateless time series data analysis service in the time series database system, the analysis operator complying with an operator rule of the time series database system, and embedding each established analysis operator in a SQL query process in the time series database system, so as to realize interaction between the time series database system and the time series data analysis system in which processing processes are isolated; the function of the analysis operator comprises: after converting time series data received from an upstream operator of the time series database system and calling information of a time series data analysis model into a JSON format analysis request and sending the analysis request to the time series data analysis system, the analysis operator is in a blocked state, after converting a JSON format analysis result received from the time series data analysis system into an analysis result conforming to an internal data format of the time series database system, the analysis operator is unblocked, and then directly or after aggregation processing, the analysis result is transmitted to a downstream operator of the time series database system; in the time series database system, the analysis operator calls a time series data analysis model adapted to the calling information through the adaptation interface in the time series data analysis system to complete time series data analysis.

2. The method of claim 1, wherein, The method comprises: separating time series data analysis functions from an existing time series database system to form an independent time series data analysis system, wherein the time series data analysis functions are implemented by time series data analysis models running in corresponding running environments.

3. The method of claim 1, wherein, The method comprises: extending, upgrading and maintaining the time series data analysis system, specifically: building a class file of the new time series data analysis model, wherein the class file of the new time series data analysis model contains an analysis method of the new time series data analysis model; saving the class file of the new time series data analysis model under a specified directory of the time series data analysis system; 4. The method of claim 1, wherein, loading the class file of the new time series data analysis model under the specified directory into the time series data analysis system, so as to extend the time series data analysis system without affecting normal operation of the time series database system. The method comprises: building an encapsulation class for each time series data analysis model in the time series data analysis system, and the encapsulation classes of the same type of time series data analysis model inherit from the same base class, so as to obtain an adaptation interface adapted to different time series data analysis models.

5. The method of claim 1, wherein, When the stateless time-series data analysis service is of the predictive analysis type, the analysis operator is a predictive analysis operator. Correspondingly, during the execution of an SQL query process in the time-series database system, the analysis operator calls the corresponding time-series data analysis model through the adaptation interface in the time-series data analysis system to complete the time-series data analysis, including: When a user calls a time-series data analysis model using SQL statements to analyze time-series data, the time-series database system executes an SQL query process. During the execution of the SQL query process, the predictive analysis operator receives the time series data output by its upstream operator and verifies the calling information of the time series data analysis model specified in the SQL statement; The predictive analysis operator converts the time series data and the calling information into a JSON format analysis request and sends it to the time series data analysis system, so that the time series data analysis system can receive and parse the JSON format analysis request, call the time series data analysis model indicated by the JSON format analysis request through the adaptation interface to perform predictive analysis, and return the JSON format analysis results of the time series data analysis model to the time series database system. The predictive analysis operator receives the JSON format analysis results returned by the time series data analysis system, converts the JSON format analysis results into the analysis results of the internal data format of the time series database system, and passes them to the downstream operator.

6. The method of claim 1, wherein, When the stateless time-series data analysis service is of the anomaly detection type, the analysis operator of the time-series data analysis system is the anomaly detection operator. Correspondingly, during the execution of an SQL query process, the analysis operator in the time-series database system calls the corresponding time-series data analysis model through the adaptation interface in the time-series data analysis system to complete the time-series data analysis, including: When a user calls a time-series data analysis model using SQL statements to analyze time-series data, the time-series database system executes an SQL query process. During the execution of the SQL query process, the anomaly detection operator receives the time-series data output by its upstream operator and verifies the call information of the time-series data analysis model specified in the SQL statement; The anomaly detection operator converts the time-series data and the call information into a JSON format analysis request and sends it to the time-series data analysis system, so that the time-series data analysis system can receive and parse the JSON format analysis request, call the time-series data analysis model indicated by the JSON format analysis request through the adaptation interface to perform anomaly detection, and return the JSON format analysis results of the time-series data analysis model to the time-series database system. The abnormality detection operator converts the received JSON format analysis result into an analysis result in the internal data format of the time series database system, and performs aggregation processing on the analysis result in the internal data format of the time series database system according to the aggregation function specified in the SQL statement, and delivers the aggregation result to the downstream operator.

7. The method of claim 1, wherein, In the execution of the SQL query process of the time series database system, before the analysis operator completes the time series data analysis by calling the time series data analysis model adapted to the calling information through the adaptation interface in the time series data analysis system, the method further comprises: adding address information of the time series data analysis system to a metadata node of the time series database system; The time series database system acquires and saves the metadata information related to the time series data analysis model provided by the time series data analysis system from the time series data analysis system according to the address information of the time series data analysis system added to the metadata node thereof.

8. The method according to any one of claims 1 to 7, characterized in that, The time series database system is deployed on a server with a CPU, and the time series data analysis system is deployed on a server with a GPU.

9. The method according to any one of claims 1-7, characterized in that, The relationship between the time series database system and the time series data analysis system is one of one-to-one, one-to-many, many-to-one, and many-to-many.

Images