Data online analysis method, system, client and server

Abstract

The application discloses a kind of data online analysis method, system, client and server, wherein the method includes: client receives the query condition input by user;According to the query condition input by user, and the relationship of pre-configured query condition and SQL statement generates SQL statement;Server creates AST object according to SQL statement;According to AST object, create the execution pipeline of query, the execution pipeline includes multiple data blocks to be queried in online analytical processing database Identification;According to multiple data blocks to be queried, match multiple data blocks to be queried from online analytical processing database, read query result from each data block to be queried, form online analysis result according to multiple query results;Online analysis result is sent to client.The application does not need data preprocessing, only needs to match corresponding data block according to the query condition configured flexibly by user, and can efficiently analyze real-time online analysis result in mass message data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, and in particular to a data online analysis method and system, a client and a server. BACKGROUND

[0002] With the exponential growth of the amount of financial field message data, the traditional online analytical processing system has extremely low preprocessing efficiency when storing a large amount of message data, and as the business needs change, the data analysis needs are also increasing, and the analysis personnel need to continuously increase the preprocessing tasks to meet the needs, the preprocessing is complex, and the real-time online analysis result cannot be obtained. SUMMARY

[0003] The embodiment of the present application provides a data online analysis method, which can efficiently analyze real-time online analysis results in massive message data without data preprocessing, only by matching corresponding data blocks according to the flexible query conditions configured by the user, and the method comprises the following steps:

[0004] The client receives the query condition input by the user, generates the SQL statement corresponding to the query condition input by the user according to the relationship between the query condition input by the user and the pre-configured query condition and SQL statement, and sends the SQL statement corresponding to the query condition input by the user to the server.

[0005] The server creates an abstract syntax tree (AST) object according to the SQL statement, creates a query execution pipeline according to the AST object, the query execution pipeline comprises a plurality of to-be-queried data block identifiers in the online analytical processing database, matches a plurality of corresponding to-be-queried data blocks from the online analytical processing database according to the plurality of to-be-queried data block identifiers, reads the query result from each to-be-queried data block, forms an online analysis result according to the query result of each to-be-queried data block, and sends the online analysis result to the client.

[0006] The embodiment of the present application provides a data online analysis method, which can efficiently analyze real-time online analysis results in massive message data without data preprocessing, only by matching corresponding data blocks according to the flexible query conditions configured by the user, and the method comprises the following steps:

[0007] Receiving the query condition input by the user;

[0008] Generating the SQL statement corresponding to the query condition input by the user according to the relationship between the query condition input by the user and the pre-configured query condition and SQL statement;

[0009] Sending the SQL statement corresponding to the query condition input by the user to the server;

[0010] Receive the online analysis result fed back according to the SQL statement corresponding to the query condition input by the user.

[0011] The embodiment of the present application provides a data online analysis method, which can efficiently analyze real-time online analysis result in mass message data without data preprocessing, by matching corresponding data block according to the query condition configured flexibly by the user, and the method comprises the following steps:

[0012] Receiving the SQL statement corresponding to the query condition input by the user;

[0013] Creating an abstract syntax tree (AST) object according to the SQL statement corresponding to the query condition input by the user;

[0014] Creating an execution pipeline of the query according to the AST object, wherein the execution pipeline of the query comprises a plurality of data block identifiers to be queried in the online analysis processing database;

[0015] Matching a plurality of data blocks to be queried corresponding to the plurality of data block identifiers to be queried from the online analysis processing database, reading query result from each data block to be queried, and forming online analysis result according to the query result of each data block to be queried;

[0016] Sending the online analysis result to the client.

[0017] The embodiment of the present application provides a data online analysis system, which can efficiently analyze real-time online analysis result in mass message data without data preprocessing, by matching corresponding data block according to the query condition configured flexibly by the user, and the system comprises the following steps:

[0018] The client is configured to receive the query condition input by the user, generate the SQL statement corresponding to the query condition input by the user according to the query condition input by the user and the relationship between the preconfigured query condition and the SQL statement, and send the SQL statement corresponding to the query condition input by the user to the server.

[0019] The server is configured to create an abstract syntax tree (AST) object according to the SQL statement, create an execution pipeline of the query according to the AST object, wherein the execution pipeline of the query comprises a plurality of data block identifiers to be queried in the online analysis processing database, match a plurality of data blocks to be queried corresponding to the plurality of data block identifiers to be queried from the online analysis processing database, read query result from each data block to be queried, form online analysis result according to the query result of each data block to be queried, and send the online analysis result to the client.

[0020] The embodiment of the present application provides a data online analysis client, which can efficiently analyze real-time online analysis results in mass message data without data preprocessing, by matching corresponding data blocks according to flexible query conditions configured by a user, and the client comprises:

[0021] A first receiving unit is configured to receive query conditions input by a user, and receive online analysis results fed back according to SQL statements corresponding to the query conditions input by the user;

[0022] A SQL statement generating unit is configured to generate the SQL statements corresponding to the query conditions input by the user according to the query conditions input by the user and a relationship between the query conditions and the SQL statements configured in advance;

[0023] A first sending unit is configured to send the SQL statements corresponding to the query conditions input by the user to a server.

[0024] The embodiment of the present application provides a data online analysis server, which can efficiently analyze real-time online analysis results in mass message data without data preprocessing, by matching corresponding data blocks according to flexible query conditions configured by a user, and the server comprises:

[0025] A second receiving unit is configured to receive the SQL statements corresponding to the query conditions input by the user;

[0026] A first creating unit is configured to create an abstract syntax tree (AST) object according to the SQL statements corresponding to the query conditions input by the user;

[0027] A second creating unit is configured to create an execution pipeline of a query according to the AST object, and the execution pipeline of the query comprises a plurality of to-be-queried data block identifiers in an online analysis processing database;

[0028] A result generating unit is configured to match a plurality of to-be-queried data blocks corresponding to the to-be-queried data block identifiers from the online analysis processing database according to the to-be-queried data block identifiers, read a query result from each to-be-queried data block, and form an online analysis result according to the query result of each to-be-queried data block;

[0029] A second sending unit is configured to send the online analysis result to the client.

[0030] The embodiment of the present application further provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the above data online analysis method when executing the computer program.

[0031] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the data online analysis method.

[0032] The embodiment of the present application also provides a computer program product, the computer program product comprises a computer program, and the computer program is executed by a processor to realize the data online analysis method.

[0033] The data online analysis scheme provided by the embodiment of the present application has the beneficial technical effects that: the client can quickly generate the SQL statement through the query condition input by the user and the relationship between the preconfigured query condition and the SQL statement, the server can only query the data block meeting the query condition by creating the abstract syntax tree AST object according to the SQL statement and creating the query execution pipeline according to the AST object, so the data meeting the condition can be quickly found from the massive data, and any data preprocessing is not needed, so that online real-time data analysis can be performed, and the real-time online data analysis result is provided.

[0034] Compared with the technical scheme that the traditional online analysis system in the prior art cannot perform real-time online data analysis with complex preprocessing and low efficiency in the massive message data analysis scene, the data online analysis scheme provided by the embodiment of the present application has the beneficial technical effects that: the client can quickly generate the SQL statement through the query condition input by the user and the relationship between the preconfigured query condition and the SQL statement, the server can only query the data block meeting the query condition by creating the abstract syntax tree AST object according to the SQL statement and creating the query execution pipeline according to the AST object, so the data meeting the condition can be quickly found from the massive data, and any data preprocessing is not needed, so that online real-time data analysis can be performed, and the real-time online data analysis result is provided. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative labor. In the drawings:

[0036] Figure 1 It is a schematic diagram of the data online analysis method applied to the system in the embodiment of the present application.

[0037] Figure 2 A bottom architecture diagram of a data online analysis service end in the embodiment of the present application;

[0038] Figure 3 A principle diagram of a data online analysis method in the embodiment of the present application;

[0039] Figure 4 A bottom architecture diagram of a data online analysis system in the embodiment of the present application;

[0040] Figure 5 A flowchart of a data online analysis method applied to a client in the embodiment of the present application;

[0041] Figure 6 A flowchart of a data online analysis method applied to a service end in the embodiment of the present application;

[0042] Figure 7 A structure diagram of a data online analysis system in the embodiment of the present application;

[0043] Figure 8 A structure diagram of a data online analysis client in the embodiment of the present application;

[0044] Figure 9 A structure diagram of a data online analysis service end in the embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, further detailed description will be given to the embodiments of the present application in combination with the drawings. Herein, the illustrative embodiments of the present application and the description thereof are used to explain the present application, but not as a limitation to the present application.

[0046] On-Line Transaction Processing (OLTP): OLTP is a computer application system taking transaction element as a unit of data processing and human-computer interaction. It can update or operate other operations on data in real time, and the data in the system is always kept in the latest state. A user can specify a sequence of operations maintaining data consistency as a transaction element, input the transaction element through a terminal, a personal computer or other devices, return the result after system processing, and apply it to airplane ticket booking, bank teller, stock trading, supermarket sales, hotel management, etc.

[0047] On-line Analytical Processing (OLAP): OLAP is a software technology that enables analysts to quickly, consistently, and interactively view information from various perspectives to gain deep insights into data. It has the feature of FASMI (Fast Analysis of Shared Multidimensional Information), where F is fast, indicating that the system can respond to most of the analysis requirements of users within seconds; A is analysis, indicating that users can define new specialized calculations as part of the analysis without programming and give reports in the way users want; M is multi-dimensional, indicating that it provides a multi-dimensional view and analysis of data analysis; and I is information, indicating that information can be obtained in time and large amounts of information can be managed.

[0048] With the exponential growth of message data in the financial field, the traditional OLTP database has been unable to meet the requirement of quickly returning the results of real-time online analysis of billions or hundreds of billions of message data due to limitations in storage mode, data compression, data indexing, query steps, aggregation mode, etc. Therefore, an embodiment of the present application proposes a data online analysis scheme, and the purpose of the scheme is to realize a new real-time online analysis system specially for large amounts of message data, so that analysts in the financial field can quickly, consistently, and interactively view message data information from various perspectives. The data online analysis scheme is described in detail below.

[0049] Figure 1 A schematic diagram of the data online analysis method applied to the system provided by the embodiment of the present application is shown in FIG. 1, and the method includes the following steps: Figure 1

[0050] Step 10: The client receives the query condition input by the user; generates the SQL statement corresponding to the query condition input by the user according to the relationship between the query condition input by the user and the pre-configured query condition and SQL statement; and sends the SQL statement corresponding to the query condition input by the user to the server.

[0051] ​Step 20: the server creates an abstract syntax tree (AST) object according to the SQL statement; creates an execution pipeline of the query according to the AST object, wherein the execution pipeline of the query comprises a plurality of data block identifiers to be queried in the online analytical processing database; matches a plurality of data blocks to be queried corresponding to the plurality of data block identifiers from the online analytical processing database; reads a query result from each data block to be queried; forms an online analysis result according to the query result of each data block to be queried; and sends the online analysis result to the client.

[0052] Specifically, in the online analysis scenario of a large amount of data messages, in order to obtain real-time online data analysis results and improve data analysis processing efficiency, the data online analysis system provided by the embodiment of the present application can include a client 01 and a server 02. Figure 7 As shown in the figure, the data online analysis system provided by the embodiment of the present application can include a client 01 and a server 02. In a specific implementation, the client 01 receives a query condition input by a user; generates a SQL statement corresponding to the query condition input by the user according to the query condition input by the user and a pre-configured relationship between the query condition and the SQL statement; sends the SQL statement corresponding to the query condition input by the user to the server; the server 02 creates an abstract syntax tree (AST) object according to the SQL statement; creates an execution pipeline of the query according to the AST object, wherein the execution pipeline of the query comprises a plurality of data block identifiers to be queried in the online analytical processing database; matches a plurality of data blocks to be queried corresponding to the plurality of data block identifiers from the online analytical processing database; reads a query result from each data block to be queried; forms an online analysis result according to the query result of each data block to be queried; and sends the online analysis result to the client.

[0053] Therefore, the data online analysis method of the embodiment of the present application supports flexible configuration of query statements, and does not require the user to encode each time according to different query conditions. Only the user needs to input a query condition, and a corresponding SQL statement can be quickly generated according to the pre-configured relationship between the query condition and the SQL statement, so as to improve the data query efficiency. Meanwhile, in the data online analysis method of the embodiment of the present application, when querying data, an AST object can be created for a SQL statement, and then an execution pipeline of the query can be quickly created according to the AST object. Only data blocks meeting the query condition can be queried, and no data preprocessing is required, so as to realize online real-time data analysis, quickly find data meeting the query condition from massive data, and improve the real-time data online analysis processing efficiency.

[0054] The online analytical processing database in the embodiment of the application is an OLAP type database, the storage structure of data in the OLAP type database is different from that of a traditional OLTP type database, and the data is stored in a block, partition and column type manner through construction of a tree type file storage structure. The data is directly written into a disk without buffering in the process of writing data, large capacity data is split into Segments of hundreds of megabytes, and is sequentially written, and the compare&merge operation is continuously performed in the process of continuous writing, so that the writing throughput is very high.

[0055] Figure 2 The figure is a bottom framework of a data online analytical service end in the embodiment of the application. The service end comprises:

[0056] 1. Data Stream: supporting reading and writing data in a medium and performing data conversion.

[0057] 2. Server: a service end interface, used for receiving data input by a client.

[0058] 3. Parser: responsible for creating an AST object and parsing each SQL statement into an AST syntax tree in a recursive descending method.

[0059] 4. Interpreter: responsible for interpreting the AST and further creating an execution pipeline of a query (that is, a query execution pipeline).

[0060] 5. Storage: an IStorage interface is responsible for definition, query and writing of data, and specifically, the IStorage interface returns original data of a specified column.

[0061] 6. Column: representing a column block in a database, a column of data is represented by a Column object, the Column object is divided into an interface and an implementation, and in an IColumn interface object, methods for performing various relational operations on data are defined. In most scenarios, data is operated in an integral column manner, but if a single specific numerical value (that is, a row of data in a single column) needs to be operated, a Field object needs to be used, and the Field object represents a single value. Unlike the generalization design idea of the Column object, the Field object uses an aggregation design mode.

[0062] 7、DataTypes: responsible for serialization and deserialization, type conversion, etc., can read and write binary form of column, thus constitute the data block in the database; IDataType interface defines a number of serialization and deserialization methods, they appear in pairs. IDataType interface also uses the generalized design pattern, the implementation logic of the specific method is carried by the instance of the corresponding data type. DataType is responsible for serialization related work, and is not responsible for data reading, but is transferred from Column or Field object.

[0063] 8、Functions: including comparison functions, sum, count, distinct functions, etc. Mainly provide two types of functions - ordinary functions (Functions) and aggregate functions (Aggregate Functions). Ordinary functions are defined by the IFunction interface, with dozens of function implementations, using vectorization to directly act on a whole column of data. Aggregate functions are defined by the IAggregateFunction interface, compared with stateless ordinary functions, aggregate functions are stateful. Take the COUNT aggregate function as an example, the state of AggregateFunctionCount uses an integer to record. The state of the aggregate function supports serialization and deserialization, so it can be transmitted between distributed nodes to realize incremental calculation.

[0064] Therefore, according to the underlying architecture of the server, the online analytical processing database adopts the data block, partition, columnar storage, data statistical organization method suitable for OLAP, and the data structure with the best performance for the specified data type.

[0065] Figure 3 The principle diagram of the data online analysis method in the embodiment of the application. Figure 4 The underlying architecture diagram of the data online analysis system in the embodiment of the application, which will be described in detail below Figure 3 and Figure 4 The data online analysis scheme of the application is described in detail.

[0066] In the specific implementation, for example Figure 3As shown, first, the client can generate dynamic SQL according to dynamic SQL configuration. Specifically, in step 10, the user can input query conditions through the client; the client can generate SQL statements corresponding to the user input query conditions according to the user input query conditions and the pre-configured relationship between the query conditions and the SQL statements (i.e. dynamic SQL configuration); thus, the query statements corresponding to the query conditions can be flexibly configured, and the code does not need to be written according to the query conditions each time, thereby improving the SQL generation efficiency and further improving the online data analysis efficiency.

[0067] In specific implementation, as shown in Figure 3 The client sends the dynamic SQL to the server of the real-time online analysis system, and the server parses the dynamic SQL. Specifically, in step 20, as shown in Figure 4 The server receives the SQL statements corresponding to the user input query conditions, and sends the SQL statements to the Parser of the server. The Parser creates an abstract syntax tree AST object according to the SQL statements. The AST object is handed over to the Interpreter for logical processing, and an execution pipeline of the query is created.

[0068] Different SQL statements perform different operations, and thus different SQL statements are parsed by different Parser analyzer implementation classes.

[0069] In one embodiment, the AST object is created according to the SQL statement, including: creating a corresponding AST object for each SQL statement corresponding to the user input query condition according to the recursive descent method;

[0070] The execution pipeline of the query is created according to the AST object, including: parsing the AST object corresponding to each SQL statement corresponding to the user input query condition to determine the execution pipeline of the corresponding query.

[0071] In specific implementation, as shown in Figure 4 The Parser parses each SQL statement into an AST syntax tree according to the recursive descent method. The AST object can be various types of structures, including queries, inserts, updates, deletions, etc. The Interpreter interprets the AST to determine the business logic corresponding to the AST object, and further creates an execution pipeline of the query (i.e. query execution pipeline) to query and convert the data at the bottom layer of the online analysis processing database.

[0072] The interpreter acts like a service layer, connecting the entire query process. It aggregates the resources needed by the interpreter based on its type.

[0073] In practice, the query execution pipeline can determine multiple data block identifiers corresponding to the AST object from the online analytical processing database. Then, when querying data, it is only necessary to match the corresponding multiple data blocks to be queried from the online analytical processing database based on the multiple data block identifiers, read the query results from each data block to be queried, and form the online analytical results based on the query results of each data block to be queried.

[0074] In one embodiment, each data block is a triple consisting of a data object, a data type, and a column identifier; the data object includes a preset relational operation function; the data type includes a preset forward serialization function and a deserialization function;

[0075] The online analytical processing (OLAP) results are generated based on the query results for each data block to be queried, including:

[0076] Retrieve the corresponding data based on the column identifier;

[0077] Call the preset relational operation function to calculate the data corresponding to the column identifier and obtain the data corresponding to the data object;

[0078] Based on the preset forward serialization and deserialization functions, the data corresponding to the data object is calculated to obtain the data corresponding to the data type;

[0079] Based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type, generate the query results corresponding to the data block to be queried.

[0080] In practice, data in the online analytical processing (OLAP) database is stored in a data block structure, consisting of a column, a data type, and a column identifier string. A series of data operations can be performed using only block objects.

[0081] In specific implementation, such as Figure 4 As shown, the IStorage interface is called through the column identifier, and the IStorage interface returns the raw data of the specified column. This interface does not process the raw data; the specific processing, calculation and filtering of the data are completed by the functions in Column and DataTypes.

[0082] In practice, the Column interface defines preset relational operation functions that can calculate the original data of a specified column to obtain the data corresponding to the data object.

[0083] In practice, the data type interface defines many forward and deserialization functions, which appear in pairs. Although DataType is responsible for serialization-related work, it does not directly handle data reading; instead, it retrieves the data from Column or Field objects. Therefore, the data type interface can obtain data from Column or Field objects, and by performing calculations on the data corresponding to the data object according to the preset forward and deserialization functions, it can obtain the data corresponding to the data type.

[0084] In practice, the query results for each data block can be generated based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type. Online analytical processing (OLAP) results are then generated based on the query results for each data block, and these OLAP results are grouped according to data object, data type, and column identifier. The OLAP results are then sent to the client.

[0085] The following describes the steps for further optimization.

[0086] In one embodiment, the client receives the expected form of the online analysis results input by the user; and sends the expected form of the online analysis results input by the user to the server.

[0087] The server receives the expected format of the online analytical results input by the user; after forming the online analytical results based on the query results of each data block to be queried, it generates the online analytical results corresponding to the expected format of the online analytical results input by the user; and sends the online analytical results corresponding to the expected format to the client.

[0088] The client receives the online analysis results corresponding to the desired format.

[0089] In specific implementation, such as Figure 3 As shown, users can also configure the result set by inputting it through the client according to their specific needs. Specifically, users input their desired format for the online analytics results (i.e., the result set configuration) through the client and send it to the server. After generating the online analytics results, the server can also generate the desired online analytics results based on the user's input format. This allows for flexible configuration of the online analytics results, eliminating the need for users to process the results each time to obtain the desired outcome, thus improving the user experience.

[0090] The advantages of the online data analysis method provided in this invention are as follows:

[0091] 1. The client can flexibly configure a query statement according to a query condition input by a user, and the user can input a desired form of online analysis result, and the server generates an online analysis result corresponding to the desired form according to the desired form of online analysis result, so that the user demand can be quickly and flexibly configured, the data online analysis processing efficiency is improved, and the user experience is improved.

[0092] 2. The server creates an AST object by processing a SQL statement, and then quickly creates an execution pipeline of the query according to the AST object, so that only data blocks meeting the query condition can be queried, and no data preprocessing is required, so that online real-time data analysis can be realized, online real-time analysis result can be provided, and the data online analysis processing efficiency is improved.

[0093] In the embodiment of the application, a data online analysis method applied to a client is also provided, as described in the following embodiment. Since the principle of solving the problem of the method is similar to that of the data online analysis method applied to the system, the implementation of the method can be referred to the implementation of the data online analysis method applied to the system, and the repeated parts will not be described again.

[0094] Figure 5 A flowchart of the data online analysis method applied to the client in the embodiment of the application is shown in FIG. 5, which includes the following steps: Figure 5

[0095] Step 501: receiving a query condition input by a user;

[0096] Step 502: generating a SQL statement corresponding to the query condition input by the user according to the query condition input by the user and a pre-configured relationship between the query condition and the SQL statement;

[0097] Step 503: sending the SQL statement corresponding to the query condition input by the user to the server;

[0098] Step 504: receiving an online analysis result fed back according to the SQL statement corresponding to the query condition input by the user.

[0099] In one embodiment, the data online analysis method applied to the client can further include:

[0100] receiving a desired form of online analysis result input by a user;

[0101] sending the desired form of online analysis result input by the user to the client;

[0102] The above step 504 can specifically include:

[0103] receiving an online analysis result corresponding to the desired form.

[0104] ​The embodiment of the present application also provides a data online analysis method applied to a server, as described in the following embodiment. Since the principle of solving problems of the method is similar to that of the data online analysis method applied to a system, implementation of the method can be referred to implementation of the data online analysis method applied to a system, and repeated parts will not be described herein.

[0105] Figure 6 A flowchart of the data online analysis method applied to a server is shown in FIG. 6, which comprises the following steps. Figure 6

[0106] Step 601: receiving a SQL statement corresponding to a query condition input by a user;

[0107] Step 602: creating an abstract syntax tree (AST) object according to the SQL statement corresponding to the query condition input by the user;

[0108] Step 603: creating an execution pipeline of a query according to the AST object, wherein the execution pipeline of the query comprises a plurality of data block identifiers to be queried in an online analytical processing database;

[0109] Step 604: matching a plurality of data blocks to be queried corresponding to the plurality of data block identifiers from the online analytical processing database according to the plurality of data block identifiers to be queried, reading a query result from each data block to be queried, and forming an online analysis result according to the query result of each data block to be queried;

[0110] Step 605: sending the online analysis result to a client.

[0111] In one embodiment, the data online analysis method applied to a server further comprises:

[0112] receiving a desired form of online analysis result input by a user;

[0113] After forming the online analysis result according to the query result of each data block to be queried, the method further comprises:

[0114] generating an online analysis result corresponding to the desired form of online analysis result input by the user;

[0115] sending the online analysis result to the client, which specifically can comprise:

[0116] sending the online analysis result corresponding to the desired form to the client.

[0117] The embodiment of the present application also provides a data online analysis system, as described in the following embodiment. Since the principle of solving problems of the system is similar to that of the data online analysis method applied to a system, implementation of the system can be referred to implementation of the data online analysis method applied to a system, and repeated parts will not be described herein.​

[0118] Figure 7 A structure diagram of a data online analytical processing system in an embodiment of the present application is shown in FIG. 1, which includes: Figure 7 A client 01 receives a user input query condition, generates a SQL statement corresponding to the user input query condition according to a pre-configured relationship between the query condition and the SQL statement, and sends the SQL statement corresponding to the user input query condition to a server.

[0119] The server 02 creates an abstract syntax tree (AST) object according to the SQL statement, creates a query execution pipeline according to the AST object, the query execution pipeline including a plurality of to-be-queried data block identifiers in an online analytical processing database, matches a plurality of corresponding to-be-queried data blocks from the online analytical processing database according to the plurality of to-be-queried data block identifiers, reads a query result from each to-be-queried data block, forms an online analytical result according to the query result of each to-be-queried data block, and sends the online analytical result to the client.

[0120] In an embodiment, the client further receives a user input online analytical result expected form, and sends the user input online analytical result expected form to the server.

[0121] The server further receives a user input online analytical result expected form, generates an online analytical result corresponding to the user input online analytical result expected form after forming the online analytical result according to the query result of each to-be-queried data block, and sends the online analytical result corresponding to the expected form to the client.

[0122] In an embodiment, the server is specifically configured to create a corresponding AST object for each user input query condition according to a recursive descent method, and parse the AST object corresponding to each user input query condition to determine a corresponding query execution pipeline.

[0123] In an embodiment, each data block is a triple composed of a data object, a data type, and a column identifier; the data object includes a pre-set relational operation function; and the data type includes a pre-set serialization function and a deserialization function.

[0124]

[0125] ​The server is specifically configured to: acquire corresponding data according to the column identifier; call a preset relationship operation function to calculate the data corresponding to the column identifier, to obtain data corresponding to the data object; calculate the data corresponding to the data object according to a preset serialization / deserialization function, to obtain data corresponding to the data type; and generate a query result corresponding to the to-be-queried data block according to the column identifier, the data corresponding to the data object and the data corresponding to the data type.

[0126] The application also provides a data online analysis client, as described in the following embodiment. Since the principle of solving the problem by the client is similar to the data online analysis method applied to the system, the implementation of the client can be referred to the implementation of the data online analysis method applied to the system, and the repeated parts will not be described again.

[0127] Figure 8 The structure diagram of the data online analysis client in the embodiment of the application is shown in FIG. 1, which includes: Figure 8

[0128] The first receiving unit 011 is configured to receive the query condition input by the user, and receive the online analysis result fed back by the SQL statement corresponding to the query condition input by the user.

[0129] The SQL statement generation unit 012 is configured to generate the SQL statement corresponding to the query condition input by the user according to the query condition input by the user and the relationship between the query condition and the SQL statement configured in advance.

[0130] The first sending unit 013 is configured to send the SQL statement corresponding to the query condition input by the user to the server.

[0131] In one embodiment, the first receiving unit is further configured to receive the expected form of the online analysis result input by the user, and receive the online analysis result corresponding to the expected form.

[0132] The first sending unit is further configured to send the expected form of the online analysis result input by the user to the server.

[0133] The application also provides a data online analysis server, as described in the following embodiment. Since the principle of solving the problem by the client is similar to the data online analysis method applied to the system, the implementation of the client can be referred to the implementation of the data online analysis method applied to the system, and the repeated parts will not be described again.

[0134] Figure 9 The structure diagram of the data online analysis server in the embodiment of the application is shown in FIG. 2, which includes: Figure 9

[0135] ​​The second receiving unit 021 is configured to receive a SQL statement corresponding to a query condition input by a user;

[0136] The first creating unit 022 is configured to create an abstract syntax tree (AST) object according to the SQL statement corresponding to the query condition input by the user;

[0137] The second creating unit 023 is configured to create an execution pipeline of a query according to the AST object, the execution pipeline of the query including a plurality of to-be-queried data block identifiers in an online analytical processing (OLAP) database;

[0138] The result generating unit 024 is configured to match a plurality of to-be-queried data blocks corresponding to the plurality of to-be-queried data block identifiers from the OLAP database according to the plurality of to-be-queried data block identifiers, read a query result from each to-be-queried data block, and form an OLAP result according to the query result of each to-be-queried data block;

[0139] The second sending unit 025 is configured to send the OLAP result to a client.

[0140] In an embodiment, the second receiving unit is further configured to receive a desired form of the OLAP result input by the user;

[0141] The result generating unit is specifically configured to generate the OLAP result in the desired form input by the user after forming the OLAP result according to the query result of each to-be-queried data block;

[0142] The second sending unit is further configured to send the OLAP result in the desired form to the client.

[0143] In an embodiment, the first creating unit is specifically configured to create the AST object corresponding to each SQL statement of the query condition input by the user according to a recursive descent method;

[0144] The second creating unit is specifically configured to parse the AST object corresponding to each SQL statement of the query condition input by the user, and determine the execution pipeline of the corresponding query.

[0145] In an embodiment, each data block is a triple composed of a data object, a data type, and a column identifier; the data object includes a preset relational operation function; and the data type includes a preset serialization function and a deserialization function in a positive order.

[0146] The result generating unit is specifically configured to obtain corresponding data according to the column identifier;

[0147] The preset relational operation function is called to calculate the data corresponding to the column identifier, and data corresponding to the data object is obtained.

[0148] According to the preset serialization function and deserialization function, the data corresponding to the data object is calculated to obtain data corresponding to the data type;

[0149] According to the column identifier, the data corresponding to the data object, and the data corresponding to the data type, a query result corresponding to the to-be-queried data block is generated.

[0150] The embodiment of the application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned data online analysis method when executing the computer program.

[0151] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned data online analysis method.

[0152] The embodiment of the application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the above-mentioned data online analysis method.

[0153] The data online analysis scheme provided by the embodiment of the application has the beneficial technical effects that: the client can quickly generate an SQL statement through the query condition input by the user and the relationship between the preconfigured query condition and the SQL statement, the server can create an abstract syntax tree (AST) object according to the SQL statement and create a query execution pipeline according to the AST object, and only the data blocks meeting the query condition can be queried, so that the data meeting the condition can be quickly found from massive data, without any data preprocessing, and online real-time data analysis can be performed to provide real-time online data analysis results.

[0154] Compared with the technical scheme of the traditional online analysis system in the prior art in the scene of analyzing a large amount of message data, the preprocessing is complex and inefficient, and real-time online data analysis cannot be performed, and the data online analysis scheme provided by the embodiment of the application has the beneficial technical effects that: the client can quickly generate an SQL statement through the query condition input by the user and the relationship between the preconfigured query condition and the SQL statement, the server can create an abstract syntax tree (AST) object according to the SQL statement and create a query execution pipeline according to the AST object, and only the data blocks meeting the query condition can be queried, so that the data meeting the condition can be quickly found from massive data, without any data preprocessing, and online real-time data analysis can be performed to provide real-time online data analysis results.

[0155] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a computer to perform any of the operations described herein. The software implementation can be for example, in the form of a computer program product. The software implementation can be implemented in a centralized fashion in one computer system or processor, or in a distributed fashion where different elements are spread across several interconnected computer systems or processors.

[0156] The present application is described in relation to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It is understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in one or more of the flow diagram or block diagram block or blocks. Figure 1 means for performing each of the functions specified in the flow diagram or block diagram block or blocks.

[0157] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagram or block diagram block or blocks. Figure 1 one or more functions specified in one or more of the flow diagram or block diagram block or blocks. Figure 1 means for performing each of the functions specified in the flow diagram or block diagram block or blocks.

[0158] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagram or block diagram block or blocks. Figure 1 one or more functions specified in one or more of the flow diagram or block diagram block or blocks. Figure 1 Figure 1 means for performing each of the functions specified in the flow diagram or block diagram block or blocks.

[0159] The specific embodiments described hereinabove are presented by way of example to teach the best application of the application. However, various modifications and changes can be made within the scope of the present application which should be understood to be limited only by the claims.

Claims

1. A method for online data analysis, characterized in that, include: The client receives the query conditions input by the user; Based on the query conditions entered by the user and the relationship between the pre-configured query conditions and the SQL statement, generate the SQL statement corresponding to the query conditions entered by the user; and send the SQL statement corresponding to the query conditions entered by the user to the server. The server creates an Abstract Syntax Tree (AST) object based on the SQL statement; Based on the AST object, a query execution pipeline is created, which includes multiple identifiers of data blocks to be queried in the online analytical processing database; Based on multiple data block identifiers, the corresponding multiple data blocks to be queried are matched from the online analytical processing database, the query results are read from each data block to be queried, and the online analytical results are formed based on the query results of each data block to be queried. The online analysis results are sent to the client. The server-side includes: Analyzer: Responsible for creating AST objects; Interpreter: Responsible for interpreting the AST and further creating the query execution pipeline; Storage abstraction: The IStorage interface is responsible for defining, querying, and writing data; Column: Represents a column block in the database; a column of data is represented by a Column object. Data types: responsible for data serialization and deserialization, and type conversion; The database's built-in function set includes comparison functions, SUM, COUNT, and DISTINCT functions. Each data block consists of a triplet comprising a data object, a data type, and a column identifier. The IStorage interface is invoked via the column identifier, returning the original data for the specified column. The Column interface defines preset relational operation functions to calculate the original data of the specified column, obtaining the data corresponding to the data object. The data type interface retrieves data from the Column object and calculates the data corresponding to the data object according to preset forward and deserialization functions, obtaining the data corresponding to the data type. The query results for the data block to be queried are generated based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type.

2. The online data analysis method as described in claim 1, characterized in that, Also includes: The client receives the expected format of the online analysis results input by the user; Send the expected format of the online analysis results input by the user to the server; The server receives the expected format of the online analysis results input by the user; After generating the online analysis results based on the query results of each data block to be queried, the online analysis results corresponding to the expected form of the online analysis results input by the user are generated. The online analysis results corresponding to the desired form are sent to the client.

3. The online data analysis method as described in claim 1, characterized in that, Based on the SQL statement, an Abstract Syntax Tree (AST) object is created, including: creating a corresponding AST object for each user-input query condition's SQL statement using a recursive descent method. Based on the AST object, a query execution pipeline is created, including parsing the AST object corresponding to the SQL statement for each user-input query condition and determining the corresponding query execution pipeline.

4. The online data analysis method as described in claim 1, characterized in that, The data object includes preset relational operation functions; the data type includes preset forward serialization functions and deserialization functions; The online analytical processing (OLAP) results are generated based on the query results for each data block to be queried, including: Retrieve the corresponding data based on the column identifier; Call the preset relational operation function to calculate the data corresponding to the column identifier and obtain the data corresponding to the data object; Based on the preset forward serialization and deserialization functions, the data corresponding to the data object is calculated to obtain the data corresponding to the data type; Based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type, generate the query results corresponding to the data block to be queried.

5. A method for online data analysis, characterized in that, include: Receive query conditions input by the user; Based on the query conditions entered by the user and the relationship between the pre-configured query conditions and the SQL statement, generate the SQL statement corresponding to the query conditions entered by the user. The SQL statement corresponding to the query conditions entered by the user is sent to the server; the server creates an Abstract Syntax Tree (AST) object based on the SQL statement. Based on the AST object, a query execution pipeline is created, which includes multiple identifiers of data blocks to be queried in the online analytical processing database; Based on multiple data block identifiers, the corresponding multiple data blocks to be queried are matched from the online analytical processing database, the query results are read from each data block to be queried, and the online analytical results are formed based on the query results of each data block to be queried. The online analysis results are sent to the client. Receive online analysis results from the SQL statements corresponding to the query conditions entered by the user; The server side includes: Analyzer: Responsible for creating AST objects; Interpreter: Responsible for interpreting the AST and further creating the query execution pipeline; Storage abstraction: The IStorage interface is responsible for defining, querying, and writing data; Column: Represents a column block in the database; a column of data is represented by a Column object. Data types: responsible for data serialization and deserialization, and type conversion; The database's built-in function set includes comparison functions, SUM, COUNT, and DISTINCT functions. Each data block consists of a triplet comprising a data object, a data type, and a column identifier. The IStorage interface is invoked via the column identifier, returning the original data for the specified column. The Column interface defines preset relational operation functions to calculate the original data of the specified column, obtaining the data corresponding to the data object. The data type interface retrieves data from the Column object and calculates the data corresponding to the data object according to preset forward and deserialization functions, obtaining the data corresponding to the data type. The query results for the data block to be queried are generated based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type.

6. The online data analysis method as described in claim 5, characterized in that, Also includes: Receive the expected format of the online analysis results input by the user; Send the expected format of the online analysis results input by the user to the client; Receiving online analysis results from SQL statements corresponding to user-input query conditions includes: receiving online analysis results corresponding to the desired format.

7. A method for online data analysis, characterized in that, Applied to the server side; including: Receive the SQL statement corresponding to the query conditions input by the user; Based on the SQL statement corresponding to the query conditions input by the user, create an Abstract Syntax Tree (AST) object; Based on the AST object, a query execution pipeline is created, which includes multiple identifiers of data blocks to be queried in the online analytical processing database; Based on multiple data block identifiers, the corresponding multiple data blocks to be queried are matched from the online analytical processing database, the query results are read from each data block to be queried, and the online analytical results are formed based on the query results of each data block to be queried. The online analysis results are sent to the client. The server-side includes: Analyzer: Responsible for creating AST objects; Interpreter: Responsible for interpreting the AST and further creating the query execution pipeline; Storage abstraction: The IStorage interface is responsible for defining, querying, and writing data; Column: Represents a column block in the database; a column of data is represented by a Column object. Data types: responsible for data serialization and deserialization, and type conversion; The database's built-in function set includes comparison functions, SUM, COUNT, and DISTINCT functions. Each data block consists of a triplet comprising a data object, a data type, and a column identifier. The IStorage interface is invoked via the column identifier, returning the original data for the specified column. The Column interface defines preset relational operation functions to calculate the original data of the specified column, obtaining the data corresponding to the data object. The data type interface retrieves data from the Column object and calculates the data corresponding to the data object according to preset forward and deserialization functions, obtaining the data corresponding to the data type. The query results for the data block to be queried are generated based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type.

8. The online data analysis method as described in claim 7, characterized in that, Also includes: Receive the expected format of the online analysis results input by the user; After generating the online analytical results based on the query results for each data block to be queried, the following are also included: Generate online analysis results corresponding to the expected format of the online analysis results input by the user; Sending the online analysis results to the client includes: The online analysis results corresponding to the desired form are sent to the client.

9. A data online analysis system, characterized in that, include: The client is used to receive query conditions input by the user; Based on the query conditions entered by the user and the relationship between the pre-configured query conditions and the SQL statement, generate the SQL statement corresponding to the query conditions entered by the user; and send the SQL statement corresponding to the query conditions entered by the user to the server. The server-side component is used to create an Abstract Syntax Tree (AST) object based on the SQL statement. Based on the AST object, a query execution pipeline is created, which includes multiple identifiers of data blocks to be queried in the online analytical processing database; Based on multiple data block identifiers, the corresponding multiple data blocks to be queried are matched from the online analytical processing database, the query results are read from each data block to be queried, and the online analytical results are formed based on the query results of each data block to be queried. The online analysis results are sent to the client. The server-side includes: Analyzer: Responsible for creating AST objects; Interpreter: Responsible for interpreting the AST and further creating the query execution pipeline; Storage abstraction: The IStorage interface is responsible for defining, querying, and writing data; Column: Represents a column block in the database; a column of data is represented by a Column object. Data types: responsible for data serialization and deserialization, and type conversion; The database's built-in function set includes comparison functions, SUM, COUNT, and DISTINCT functions. Each data block consists of a triplet comprising a data object, a data type, and a column identifier. The IStorage interface is invoked via the column identifier, returning the original data for the specified column. The Column interface defines preset relational operation functions to calculate the original data of the specified column, obtaining the data corresponding to the data object. The data type interface retrieves data from the Column object and calculates the data corresponding to the data object according to preset forward and deserialization functions, obtaining the data corresponding to the data type. The query results for the data block to be queried are generated based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type.

10. The online data analysis system as described in claim 9, characterized in that, The client is also used to receive the expected form of the online analysis results input by the user; and to send the expected form of the online analysis results input by the user to the server. The server is also used to receive the user's input of the expected format of the online analysis results; After generating the online analysis results based on the query results of each data block to be queried, the online analysis results corresponding to the expected form of the online analysis results input by the user are generated. The online analysis results corresponding to the desired form are sent to the client.

11. The online data analysis system as described in claim 9, characterized in that, Specifically, the server is used to: create a corresponding AST object for the SQL statement corresponding to each user's input query conditions using a recursive descent method; parse the AST object corresponding to the SQL statement corresponding to each user's input query conditions to determine the execution pipeline of the corresponding query.

12. The online data analysis system as described in claim 9, characterized in that, The data object includes preset relational operation functions; the data type includes preset forward serialization functions and deserialization functions; Specifically, the server is used to: retrieve the corresponding data based on the column identifier; call a preset relational operation function to calculate the data corresponding to the column identifier to obtain the data corresponding to the data object; calculate the data corresponding to the data object based on a preset forward and deserialization function to obtain the data corresponding to the data type; and generate the query results corresponding to the data block to be queried based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type.

13. A data online analysis client, characterized in that, include: The first receiving unit is used to receive the query conditions input by the user; Receive online analysis results from the SQL statements corresponding to the query conditions entered by the user; The SQL statement generation unit is used to generate the SQL statement corresponding to the query conditions input by the user, based on the query conditions input by the user and the relationship between the pre-configured query conditions and the SQL statement. The first sending unit is used to send the SQL statement corresponding to the query conditions input by the user to the server; the server creates an Abstract Syntax Tree (AST) object based on the SQL statement. Based on the AST object, a query execution pipeline is created, which includes multiple identifiers of data blocks to be queried in the online analytical processing database; Based on multiple data block identifiers, the corresponding multiple data blocks to be queried are matched from the online analytical processing database, the query results are read from each data block to be queried, and the online analytical results are formed based on the query results of each data block to be queried. The online analysis results are sent to the client. The server side includes: Analyzer: Responsible for creating AST objects; Interpreter: Responsible for interpreting the AST and further creating the query execution pipeline; Storage abstraction: The IStorage interface is responsible for defining, querying, and writing data; Column: Represents a column block in the database; a column of data is represented by a Column object. Data types: responsible for data serialization and deserialization, and type conversion; The database's built-in function set includes comparison functions, SUM, COUNT, and DISTINCT functions. Each data block consists of a triplet comprising a data object, a data type, and a column identifier. The IStorage interface is invoked via the column identifier, returning the original data for the specified column. The Column interface defines preset relational operation functions to calculate the original data of the specified column, obtaining the data corresponding to the data object. The data type interface retrieves data from the Column object and calculates the data corresponding to the data object according to preset forward and deserialization functions, obtaining the data corresponding to the data type. The query results for the data block to be queried are generated based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type.

14. The online data analysis client as described in claim 13, characterized in that, The first receiving unit is further configured to receive the user's input of the expected form of the online analysis result; and to receive the online analysis result corresponding to the expected form; The first sending unit is also used to send the expected form of the online analysis result input by the user to the server.

15. A data online analysis server, characterized in that, include: The second receiving unit is used to receive the SQL statement corresponding to the query conditions input by the user; The first creation unit is used to create an Abstract Syntax Tree (AST) object based on the SQL statement corresponding to the query conditions input by the user. The second creation unit is used to create a query execution pipeline based on the AST object. The query execution pipeline includes multiple data block identifiers to be queried in the online analytical processing database. The result generation unit is used to match multiple corresponding data blocks to be queried from the online analytical processing database based on multiple data block identifiers, read the query results from each data block to be queried, and form online analytical results based on the query results of each data block to be queried. The second sending unit is used to send the online analysis results to the client; The server-side includes: Analyzer: Responsible for creating AST objects; Interpreter: Responsible for interpreting the AST and further creating the query execution pipeline; Storage abstraction: The IStorage interface is responsible for defining, querying, and writing data; Column: Represents a column block in the database; a column of data is represented by a Column object. Data types: responsible for data serialization and deserialization, and type conversion; The database's built-in function set includes comparison functions, SUM, COUNT, and DISTINCT functions. Each data block consists of a triplet comprising a data object, a data type, and a column identifier. The IStorage interface is invoked via the column identifier, returning the original data for the specified column. The Column interface defines preset relational operation functions to calculate the original data of the specified column, obtaining the data corresponding to the data object. The data type interface retrieves data from the Column object and calculates the data corresponding to the data object according to preset forward and deserialization functions, obtaining the data corresponding to the data type. The query results for the data block to be queried are generated based on the column identifier, the data corresponding to the data object, and the data corresponding to the data type.

16. The online data analysis server as described in claim 15, characterized in that, The second receiving unit is also used to receive the user's input of the expected form of the online analysis results; The result generation unit is specifically used to generate online analysis results corresponding to the expected form of the online analysis results input by the user after forming online analysis results based on the query results of each data block to be queried; The second sending unit is also used to send the online analysis results corresponding to the expected form to the client.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images