Data query method and device, computer device, and storage medium

Abstract

The application relates to a data query method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining a first type of query statement, determining a syntax expression and an expected data format corresponding to the first type of query statement; based on a target database in which data queried by the first type of query statement is located, constructing a second type of query statement matched with the target database according to data query content represented by the syntax expression; executing the second type of query statement to perform data query on the target database, obtaining target data in a target data format, and the target data format being matched with the second type of query statement; and performing data format conversion on the target data according to data conversion logic between the expected data format and the target data format, and obtaining a query result matched with the first type of query statement. The method can realize convenient and accurate query on data stored in the target database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data query method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] With the development of computer technology, data storage and data query are very important in the data monitoring process. However, due to the wide distribution range and large amount of data required to be monitored by the data monitoring system, the database of the data monitoring system itself cannot realize the storage of the huge data, which limits the data monitoring process.

[0003] In order to solve the problem of limited data storage, a processing scheme of using a third-party database for data storage is proposed. However, the data storage space provided by the third-party database is subject to the data storage mode of the third-party database, and the monitoring system cannot realize convenient and accurate query of the data stored in the third-party database. SUMMARY

[0004] Therefore, it is necessary to provide a data query method and device, computer equipment, computer readable storage medium and computer program product which can realize convenient and accurate query.

[0005] In a first aspect, the present application provides a data query method. The method comprises:

[0006] obtaining a first type of query statement, determining a syntax expression and an expected data format corresponding to the first type of query statement;

[0007] based on the target database where the data queried by the first type of query statement is located, constructing a second type of query statement matched with the target database according to the data query content represented by the syntax expression;

[0008] executing the second type of query statement to query the target database, obtaining target data in a target data format, and the target data format matches the second type of query statement;

[0009] performing data format conversion on the target data according to the data conversion logic between the expected data format and the target data format, and obtaining a query result matched with the first type of query statement.

[0010] In a second aspect, the present application further provides a data query device. The device comprises:

[0011] a statement obtaining module, configured to obtain a first type of query statement, and determine a syntax expression and an expected data format corresponding to the first type of query statement;

[0012] constructing a second type of query statement which is adapted to the target database according to the data query content represented by the syntax expression based on the target database where the data queried by the first type of query statement is located;

[0013] performing the second type of query statement to query the target database and obtaining target data in a target data format which matches the second type of query statement;

[0014] performing data format conversion on the target data according to data conversion logic between the expected data format and the target data format to obtain a query result which matches the first type of query statement.

[0015] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] obtaining a first type of query statement, determining a syntax expression and an expected data format corresponding to the first type of query statement;

[0017] constructing a second type of query statement which is adapted to the target database according to the data query content represented by the syntax expression based on the target database where the data queried by the first type of query statement is located;

[0018] performing the second type of query statement to query the target database and obtaining target data in a target data format which matches the second type of query statement;

[0019] performing data format conversion on the target data according to data conversion logic between the expected data format and the target data format to obtain a query result which matches the first type of query statement.

[0020] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0021] obtaining a first type of query statement, determining a syntax expression and an expected data format corresponding to the first type of query statement;

[0022] constructing a second type of query statement which is adapted to the target database according to the data query content represented by the syntax expression based on the target database where the data queried by the first type of query statement is located;

[0023] execute the second type of query statement to perform data query on the target database to obtain target data in a target data format, the target data format being matched with the second type of query statement;

[0024] perform data format conversion on the target data according to data conversion logic between the expected data format and the target data format to obtain a query result matched with the first type of query statement.

[0025] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the following steps:

[0026] obtain a first type of query statement, determine a syntax expression and an expected data format corresponding to the first type of query statement;

[0027] based on a target database where data queried by the first type of query statement is located, construct a second type of query statement matched with the target database according to data query content represented by the syntax expression;

[0028] execute the second type of query statement to perform data query on the target database to obtain target data in a target data format, the target data format being matched with the second type of query statement;

[0029] perform data format conversion on the target data according to data conversion logic between the expected data format and the target data format to obtain a query result matched with the first type of query statement.

[0030] The above data query method, device, computer device, storage medium and computer program product, by obtaining a first type of query statement, determining a syntax expression and an expected data format corresponding to the first type of query statement, constructing a second type of query statement matched with a target database where data queried by the first type of query statement is located according to data query content represented by the syntax expression, realizing conversion of query statements, facilitating data query, performing data query on the target database by executing the second type of query statement, realizing obtaining of stored data to obtain target data in a data format matched with the second type of query statement, and performing data format conversion on the target data according to data conversion logic between the expected data format and the target data format to make the query result matched with the first type of query statement, realizing convenient and accurate query of stored data in the database. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 a diagram of an application environment of a data query method in an embodiment;

[0032] Figure 2A flowchart of a data query method in an embodiment;

[0033] Figure 3 A flowchart of query statement reconstruction in an embodiment;

[0034] Figure 4 A flowchart of constructing a syntax expression based on a structured syntax tree in an embodiment;

[0035] Figure 5 A flowchart of data reconstruction and query in a data query method in an embodiment;

[0036] Figure 6 An interaction diagram of a query object and a target database in an embodiment;

[0037] Figure 7 A structural diagram of a data query system in an embodiment;

[0038] Figure 8 A flowchart of a data query method in an embodiment;

[0039] Figure 9 A flowchart of data storage in a target database in an embodiment;

[0040] Figure 10 A structural diagram of a data query system in another embodiment;

[0041] Figure 11 A structural block diagram of a data query device in an embodiment;

[0042] Figure 12 An internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0044] Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or a local area network to realize data calculation, storage, processing and sharing.

[0045] Cloud technology (network technology, information technology, integration technology, management platform technology, application technology based on cloud computing business model application, can constitute a resource pool, on demand, flexible and convenient. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portals. With the high development and application of the Internet industry, every item in the future may have its own identification mark and needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support, which can only be realized through cloud computing.

[0046] Database (Database), in a nutshell, can be regarded as an electronic file cabinet - a place to store electronic files, users can add, query, update, delete and other operations on the data in the file. So-called "database" is a collection of data stored together in a certain way, can be shared by multiple users, has the smallest possible redundancy, and is independent of application programs.

[0047] The data query method provided by the embodiments of the application can be applied to an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other servers. Taking a monitoring system running on the terminal 102 and displaying visual monitoring results as an example, the monitoring system sends a first type of query statement compatible with the monitoring system to the server 104. The server 104 obtains the first type of query statement, determines the syntax expression and the expected data format corresponding to the first type of query statement, constructs a second type of query statement compatible with the target database based on the target database where the data queried by the first type of query statement is located, according to the data query content represented by the syntax expression, executes the second type of query statement, and performs data query on the target database to obtain target data in a target data format, the target data format being matched with the second type of query statement; according to the data conversion logic between the expected data format and the target data format, the target data is converted in data format to obtain a query result matched with the first type of query statement, and the query result is fed back to the monitoring system for monitoring result display.

[0048] The terminal 102 can be, but is not limited to, various desktop computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. The server 104 can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0049] In one embodiment, as shown in Figure 2 , a data query method is provided, which can be applied to a computer device. The above method is taken as an example to illustrate the server in Figure 1 , which includes the following steps:

[0050] In step 202, a first type of query statement is obtained, and a syntax expression corresponding to the first type of query statement and an expected data format are determined.

[0051] The query statement is a database query and a program design language, which is used for data storage, query, and management operations on a database. The database is a data space for data storage according to certain data storage rules. Different databases can have different data storage rules, and the data query in different databases can be based on a query language with corresponding syntax. For example, database A stores data according to the first type of rule, and can query data in database A according to the A type of query statement. Database B stores data according to the second type of rule, and can query data in database B according to the B type of query statement.

[0052] The statement category of the first type of query statement is adapted to the query object that issues the query statement. The query object can be a client that interacts with a user, such as a data query system, a data query module in a data monitoring system, etc. The first type of query statement can be a statement directly input by the user through the client, or a statement generated by the client in response to a query operation triggered by the user.

[0053] The syntax expression is a combination of numbers, operators, number grouping symbols, free variables and constraint variables in a meaningful arrangement method that can obtain a numerical value. The syntax expression can represent the data range required by the first type of query statement, i.e., the data query content. Specifically, the syntax expression can be a function expression, the input of which is the fixed value of the free variable, and the output of which is the result obtained after calling the function to find data.

[0054] The expected data format is the data format of the data query result matched with the data processing function of the first type of query statement. The expected data format is determined based on the category and function of the query statement. For example, the A type query statement has a data aggregation function, and the expected data format of the query result corresponding to the A type query statement is aggregated data. For another example, the B type query statement has a data statistical function, and the expected data format of the query result corresponding to the A type query statement is statistical data.

[0055] Specifically, the computer device sends the first type of query statement matched with the query object to the server, the server performs syntax and function analysis on the obtained first type of query statement, and determines the syntax expression and the expected data format corresponding to the first type of query statement. The syntax analysis and function analysis can be synchronous processing processes, or asynchronous processing processes, and do not interfere with each other. The server obtains the syntax expression of the first type of query statement by performing syntax analysis on the first type of query statement, and determines the expected data format of the first type of query statement by performing function analysis on the first type of query statement. Further, the function analysis can determine the expected data format of the first type of query statement through the statement category of the first type of query statement, and through the preset association relationship between the query statement category and the expected data format.

[0056] Step 204, based on the target database where the data queried by the first type of query statement is located, constructing a second type of query statement matched with the target database according to the data query content represented by the syntax expression.

[0057] The target database is a database for storing data queried by the first type of query statement. The data queried by a query statement is stored in a specified database, and the specified database can be a single database or multiple databases of the same category. Databases of the same category support the same query statement for data query. The target database can be one of a columnar database, a time series database, etc. The columnar database is a database that stores data in a column-related storage architecture and is mainly suitable for batch data processing and instant query, such as a ClickHouse database. The time series database is a short name for a time series database, which refers to a database mainly used for processing data with time labels. Each data changes in time order, i.e., time serialization, and the data with time labels is time series data, such as TSDB (a time series database name), CTSDB (a time series database name), etc.

[0058] The query statement or the query request including the query statement received by the server can carry a database identifier, which is used to represent the target database where the queried data is located. The target database can be a database in the server that obtains the first type of query statement, or a database in a second server that interacts with the first server that obtains the first type of query statement.

[0059] The second type of query statement is a query statement adapted to the target database, and the second type of query statement can be used to query the data stored in the target database. It should be noted that the first type of query statement and the second type of query statement are intended to refer to different types of query statements, and are not limited to specific query statements. In other embodiments, the types of query statements referred to by the first type of query statement and the second type of query statement can be interchangeable.

[0060] Specifically, the server receives a query request initiated by a query object, extracts a database identifier carried in the query request and a first type of query statement adapted to the query object, determines a target database where data queried by the first type of query statement is located based on the database identifier, and constructs a second type of query statement belonging to a target query statement category based on the target query statement category adapted to the target database and according to the data query content represented by the syntax expression of the first type of query statement.

[0061] In step 206, the second type of query statement is executed to query the target database and obtain target data in a target data format. The target data format matches the second type of query statement.

[0062] The target data is data obtained from the database, the target data format is a data format matched with the second type of query statement, the target data format and the expected data format belong to different data format categories, and data format conversion can be performed between data of the target data format and data of the expected data format.

[0063] Specifically, the data format conversion between the target data format and the expected data format can be one-way conversion, that is, the server can convert data of the target data format into data of the expected data format, but cannot reversely convert data of the expected data format into data of the target data format. The data format conversion between the target data format and the expected data format can be two-way conversion, that is, the server can convert data of the target data format into data of the expected data format, and can also reversely convert data of the expected data format into data of the target data format.

[0064] In a specific application, taking the server obtaining the first type of query statement and the server where the target database is located as the same server as an example, the server executes the second type of query statement, performs data query on the target database based on the syntax characteristics of the second type of query statement, and obtains target data in a target data format. When the target data format corresponding to the second type of query data is an original storage format, the server executes the second type of query statement, directly extracts original data meeting the query condition corresponding to the second type of query statement from the target database, that is, the target data. When the target data format corresponding to the second type of query data is a non-original storage format, the server executes the second type of query statement, determines original data meeting the query condition corresponding to the second type of query statement from the target database, and converts the original data into target data in the target data format.

[0065] Step 208: performing data format conversion on the target data according to data conversion logic between the expected data format and the target data format, to obtain a query result matched with the first type of query statement.

[0066] The data conversion logic is data processing logic for converting data of the target data format into data of the expected data format. The data conversion logic can be preconfigured data processing logic. Data format conversion refers to a processing process of converting data in a current data format into data in a specified data format, and the specific processing principle of data format conversion can be implemented according to the data conversion logic.

[0067] Specifically, after determining the expected data format corresponding to the first type of query statement and the target data format corresponding to the second type of query statement, the server finds the data conversion logic between the expected data format and the target data format from the configuration data, with the target data format being the format before conversion and the expected data format being the format after conversion as the search target. The server converts the target data in the target data format into data in the expected data format according to the data conversion logic, which is the query result matched with the first type of query statement.

[0068] The above data query method determines the syntax expression and the expected data format corresponding to the first type of query statement by obtaining the first type of query statement, constructs a second type of query statement matched with the target database where the data queried by the first type of query statement is located according to the data query content represented by the syntax expression, realizes conversion of the query statement, facilitates data query, performs data query on the target database by executing the second type of query statement, and obtains target data matched with the second type of query statement in terms of data format, thereby realizing acquisition of the stored data, converting the target data according to the data conversion logic between the expected data format and the target data format, so that the query result can be matched with the first type of query statement, and realizing convenient and accurate query of the stored data in the database.

[0069] In one embodiment, obtaining the first type of query statement and determining the syntax expression of the first type of query statement include: performing syntax analysis on the obtained first type of query statement to determine the syntax hierarchical architecture of the first type of query statement; and constructing the syntax expression of the first type of query statement according to the syntax hierarchical architecture.

[0070] The syntax analysis is a process of analyzing the syntax structure of the query statement according to certain syntax rules and giving a structural expression of the query statement. The syntax analysis involves the expression and storage method of the query statement syntax in the computer and the algorithm of the syntax analysis. The process of the syntax analysis can be based on the expression symbols in the query statement to perform hierarchical peeling, thereby obtaining the associated relationship between the hierarchical levels of the query statement.

[0071] The syntax hierarchical architecture is the associated relationship between the hierarchical levels after hierarchical peeling of the query statement, wherein the associated relationship between different hierarchical levels can include a nested relationship and a parallel relationship. The data corresponding to two hierarchical levels with a nested relationship is an upper level and a lower level, for example, the relationship between a folder and a subfolder in a database is a nested relationship. The data corresponding to two hierarchical levels with a parallel relationship is the same level, for example, the relationship between two subfolders belonging to the same folder in a database is a parallel relationship, and for another example, the relationship between different column data in a data table is also a parallel relationship. The syntax hierarchical architecture can be represented by a syntax tree or a structure table.

[0072] Specifically, the server performs syntax analysis on the obtained first type of query statement through hierarchical separation to obtain a syntax hierarchical architecture of the first type of query statement and a syntax structure body of each level, so as to express the relationship between each level through the syntax hierarchical architecture. The server nests and collects the syntax structure body of each level according to the relationship between each level to obtain a syntax expression of the first type of query statement.

[0073] In a specific application, the first type of query statement input is a text. The server converts the text into a structured syntax tree object by analyzing the text, and then implements corresponding data calculation logic. Specifically, when the semantics expressed by the first type of query statement is to count the average number of triggers of the same instance triggered in parallel in a monitoring system within a preset time range, the syntax element can be divided into three layers of meanings. The first layer can represent that the monitoring object of the statistics is an instance, the second layer is the statistical attribute, that is, the average number of triggers, and the third layer defines the position of the monitoring object and the monitoring time range. The nested relationship between the contents of each level after analysis is the syntax expression of the first type of query statement. Based on the nested relationship and the meanings of each level, a second type of query statement with different expression modes can be obtained. Corresponding to the semantics expressed by the first type of query statement, the semantics expressed by the second type of query statement can be to filter the instances triggered in parallel in the monitoring system within a time range from a specified start time stamp.

[0074] In a specific application, as Figure 3As shown, taking the code syntax of the first type of query statement as sum(avg_over_time(go_goroutines{job="prometheus"}[5m]))by(instance) as an example, the meaning expressed thereby is to count the number of instances of parallel triggering in the prometheus system within 5 minutes. The first type of query statement can be decomposed from outside to inside into three layers: the first layer is sum(…)by(instance), which indicates that instances that are the same are allocated to a group; the second layer is avg_over_time(…), which indicates counting the average number of triggering times; and the third layer is go_goroutines{job="prometheus"}[5m], which indicates monitoring the concurrent processes in prometheus and the time range for counting is 5 minutes. The second type of query statement constructed is Select::field form go_goroutines where job="prometheus" and timestamp>t-5m and timestamp<t. In fact, based on the nested relationship of the second type of query statement, the content expressed by each layer of the first type of query statement is reorganized. The meaning actually expressed by the second type of query statement is to filter the instances of parallel triggering in prometheus from the time range of t-5m to t. The second type of query statement uses a different query statement, but the filtering object expressed thereby is the same.

[0075] In this embodiment, by performing syntax analysis on the first type of query statement, the syntax hierarchical architecture of the first type of query statement is determined, so as to clearly express the relationship between each layer by analysis, and to realize quick and accurate analysis of the first query statement according to the syntax hierarchical architecture, so as to facilitate reconstruction of the query statement and improve data processing efficiency.

[0076] In one embodiment, the syntax expression includes an instantaneous expression or a range expression. According to the syntax hierarchical architecture, the syntax expression of the first type of query statement is constructed, including: converting the first type of query statement into a structured syntax tree according to the syntax hierarchical architecture; when the structured syntax tree is a single-layer syntax tree, determining an instantaneous expression corresponding to the first type of query statement; and when the structured syntax tree is a multi-layer syntax tree, traversing each layer of the structured syntax tree to obtain a range expression corresponding to the first type of query statement.

[0077] The structured syntax tree is a tree form of the syntax structure of the source code. Different languages correspond to different syntax analyzers. The syntax analyzer is a program that reads the source code as a string, parses it, and establishes a syntax tree. The design of the syntax and the implementation of the syntax analyzer are important factors in determining the external representation of the language. Specifically, the structured syntax tree of the first type of query statement can be a function call relationship tree. Each node is a function, and the child nodes are the parameters of the function. The type of the parameter can be a function itself.

[0078] The instantaneous expression refers to an expression whose return value contains only the latest sample value in the time series. For example, in a set of time series, each of which contains a single sample, the set of time series shares the same timestamp, i.e., the timestamp corresponding to the single sample. When the structured syntax tree is a single-layer syntax tree, i.e., the structured syntax tree does not have a nested hierarchical structure, based on the structure of the single-layer syntax tree, the instantaneous expression corresponding to the first type of query statement is determined.

[0079] The range expression, also known as the interval expression, refers to an expression whose return value contains sample data within a time range. When the structured syntax tree is a multi-level syntax tree, i.e., the structured syntax tree includes a nested hierarchical structure, based on the structure of the multi-level syntax tree, each level of the structured syntax tree is traversed layer by layer to obtain the range expression corresponding to the first type of query statement.

[0080] Specifically, the server performs syntax analysis on the obtained first type of query statement to obtain the syntax hierarchical architecture of the first type of query statement and the syntax structure of each level. The syntax structure is the result of hierarchical splitting according to the syntax structure of the statement. Then, the server aggregates the syntax structure of each level according to the syntax hierarchical architecture, converts the first type of query statement into a structured syntax tree, and determines the expression type and the syntax expression corresponding to the first type of query statement according to the number of levels of the structured syntax tree. Specifically, when the structured syntax tree is a single-layer syntax tree, the server determines the instantaneous expression corresponding to the first type of query statement based on the structure of the single-layer syntax tree. When the structured syntax tree is a multi-level syntax tree, the server traverses the syntax structure of each level of the structured syntax tree layer by layer to construct the range expression corresponding to the first type of query statement.

[0081] In this embodiment, by converting the first type of query statement into a structured syntax tree, using the intuitive expression of the structured syntax tree, and quickly and accurately determining the expression type and the syntax expression corresponding to the first type of query statement according to the number of levels of the structured syntax tree, the data processing flow for determining the syntax expression can be effectively reduced, the occupation of data processing resources can be reduced, and the data processing speed can be improved.

[0082] In one embodiment, the range expression corresponding to the first type of query statement is obtained by traversing each level of the structured syntax tree, including: determining, for each level of the structured syntax tree, a sub-expression represented by the level; and aggregating the sub-expressions of each level to obtain the range expression corresponding to the first type of query statement.

[0083] The sub-expression is a part of a complete expression, and each sub-expression can be used as an independent element. The sub-expressions are allowed to be nested. The structure is a structured data composed of a plurality of data, and each data constituting the structured data is referred to as a member of the structured data. The structure of each level of the structured syntax tree is obtained by processing the result of splitting the first type of query statement according to the hierarchical structure according to the syntax rule of the syntax tree.

[0084] Specifically, in the multi-level syntax tree, the structure of the next level is part of the structure of the previous level. The sub-expression corresponding to the syntax structure of the next level and the sub-expression corresponding to the syntax structure of the previous level are in a nested relationship. The sub-expressions of each level can be aggregated by at least one of the processing modes such as nesting and merging, thereby obtaining the range expression corresponding to the first type of query statement.

[0085] In this embodiment, for the multi-level syntax tree, the data processing of each level is independently performed by determining the sub-expression of each level, the determination process of the expression is split into multiple independent steps, and each step can be performed synchronously, thereby improving the data processing efficiency and quickly obtaining the range expression corresponding to the multi-level syntax tree.

[0086] In one embodiment, the sub-expression includes a vector expression or a scalar expression. For each level of the structured syntax tree, the sub-expression represented by the level is determined, including: determining, for each level of the structured syntax tree, an expected data type of the level; when the expected data type is vector data, determining a vector expression of the level; and when the expected data type is scalar data, determining a scalar expression of the level.

[0087] The expected data type refers to the data type of the return result that can be obtained, and includes vector data and scalar data. Specifically, the vector data is time series data, and the vector data in the target database can be obtained by querying the vector expression; and the scalar data is a floating-point data value or a string value. The scalar data in the target database can be obtained by querying the scalar expression.

[0088] In one specific application, based on the structured syntax tree, the data processing procedure of obtaining the syntax expression of the first type of syntax data is as shown in Figure 4 As shown in the figure, for the structured syntax tree, the server first determines whether it is a transient query based on the number of levels of the structured syntax tree. If the structured syntax tree is a single-level syntax tree, i.e., the first type of syntax data is a transient query, then a transient expression is directly constructed based on the single-level syntax tree. If the structured syntax tree is a multi-level syntax tree, i.e., the first type of syntax data is not a transient query, then it is determined that the expression to be constructed is a range expression. Each level of the multi-level syntax tree is traversed, and the processing procedure for each level includes: determining whether the expected data type of the structure of the level is scalar data or vector data based on the structure of the level. If it is scalar data, then a scalar sub-expression or a floating-point sub-expression is constructed. If it is vector data, then a vector expression is constructed. When the traversal of each level of the multi-level syntax tree is completed, the range expression corresponding to the first type of syntax data is obtained.

[0089] In this embodiment, the server determines the expected data type of each level of the structured syntax tree based on the structure of the level, which can construct the expression according to the expression type in a targeted manner to ensure the accuracy of the obtained query data.

[0090] In one embodiment, the sub-expressions of each level are aggregated to obtain the range expression corresponding to the first type of query statement, including: aggregating the sub-expressions of each level to determine the data time range and data label to be queried; and constructing the range expression corresponding to the first type of query statement according to the data time range and data label.

[0091] The data in the target database is time series data stored in the order of time stamp and value sequence. The data label of each time series data includes an index name and a set of label sets. The time series data consists of an index, a time stamp, and a sample value. The data label includes an index name and a label set describing the characteristics of the data. The index name is used to reflect the meaning of the data, which can be composed of ASCII characters, numbers, underscores, and colons and comply with regular expressions. Each label in the label set is used to reflect the characteristic dimension of the data, and the query data can be filtered, aggregated, etc. through the characteristic dimension.

[0092] Specifically, the server aggregates the sub-expressions of each level to determine the data time range and data label to be queried based on the data structure of the data stored in the target database, thereby locking the data range to be queried in the database. The range expression corresponding to the first type of query statement is constructed according to the data time range and data label, so that the range expression matches the data structure in the target database, facilitating the further construction of the second type of query statement that is adapted to the target database.

[0093] In the embodiment, the range expression corresponding to the first type of query statement is constructed by determining the data time range and data label to be queried, which can make the range expression match the data structure in the target database, ensure the adaptation of the second type of query statement to the target database, and improve the accuracy of the data query result.

[0094] In one embodiment, based on the target database where the data queried by the first type of query statement is located, the second type of query statement adapted to the target database is constructed according to the data query content represented by the syntax expression, including: determining the statement category adapted to the target database based on the target database where the data queried by the first type of query statement is located; and constructing the second type of query statement based on the data time and data label represented by the syntax expression according to the statement construction mode corresponding to the statement category.

[0095] The category of the query statement used for data query of the target database can be one, or two or more. When the category of the query statement used for data query of the target database is one, the statement category adapted to the target database is the query statement of this category. When the category of the query statement used for data query of the target database is two or more, the statement category adapted to the target database can be the unique target category of the query statement determined from the two or more categories of the query statement. The target category can be the query statement category with the highest adaptation degree to the target database, or the query statement category with the fastest query efficiency, which can be set according to the actual application scenario. By determining the statement category of the query statement, the second type of query statement constructed has a high adaptation degree to the target database, which is convenient for subsequent data query of the target database.

[0096] The statement construction mode corresponds to the statement category, and the statement construction modes corresponding to different statement categories can be different. By determining the statement category adapted to the target database, the second type of query statement constructed based on the statement construction mode corresponding to the statement category can be used for data query in the target database.

[0097] Specifically, the server determines the target statement category adapted to the target database based on the target database where the data queried by the first type of query statement is located, based on the matching relationship between the database and the statement category of the query statement, and constructs the second type of query statement based on the data time and data label represented by the syntax expression according to the statement construction mode corresponding to the target statement category, so as to query the time series data in the target database based on the second type of query statement adapted to the target database.

[0098] In the embodiment, the data time and the data label represented by the syntax expression construct the second type of query statement, which can match the second type of query statement with the data structure in the target database, and improve the accuracy of the data query result.

[0099] In one embodiment, the data in the expected data format includes aggregated data, the data in the target data format includes sparse data, and the data format conversion of the target data according to the data conversion logic between the expected data format and the target data format obtains the query result matched with the first type of query statement, including: according to the data category of the expected aggregated data, the sparse data belonging to the data category is aggregated to obtain the query result matched with the first type of query statement.

[0100] The aggregated data refers to the data obtained by performing data aggregation on the original data, and the sparse data is relative to the aggregated data. Specifically, the sparse data can be the original data queried from the target database, and the result obtained by performing aggregation on the sparse data is the aggregated data.

[0101] Specifically, the server aggregates the sparse data belonging to the data category according to the data category of the expected aggregated data. When the data category of the expected aggregated data includes two or more categories, the sparse data belonging to each target data category is aggregated respectively to obtain the query result of multiple data categories.

[0102] Specifically, as shown in FIG. 4, the server aggregates the sparse data belonging to the data category according to the data category of the expected aggregated data. Figure 5 and Figure 6As shown, the server receives a first type of query statement sent by a query object, calculates a step length of data queried by the first type of query statement, when the step length meets a query condition, performs expression parsing on the first type of query statement, obtains a structured syntax tree, and then determines whether it is a transient query based on a number of levels of the structured syntax tree. If the structured syntax tree is a single-level syntax tree, i.e., the first type of syntax data is a transient query, a transient expression is directly constructed based on the single-level syntax tree. If the structured syntax tree is a multi-level syntax tree, i.e., the first type of syntax data is not a transient query, it is determined that a range expression needs to be constructed. Each level of the multi-level syntax tree is traversed, and the processing procedure of each level includes: determining whether the expected data type of the level is scalar data or vector data based on the structure of the level, constructing a scalar sub-expression or a floating-point sub-expression if it is scalar data, and constructing a vector expression if it is vector data. When the traversal of each level of the multi-level syntax tree is completed, a range expression corresponding to the first type of syntax data is obtained. For the range expression or the transient expression, the time range, the index, and the label value corresponding to the expression are calculated, a second type of query statement compatible with a target database is constructed based on the time range, the index, and the label value, data query is performed based on the second type of query statement, query data is obtained, aggregation operation is performed again on the query data because the expected data format of the first type of query statement is aggregated data, aggregated results are obtained, and the aggregated results are fed back to the query object.

[0103] In a specific application, the first type of query statement is a PromQL (Prometheus Query Language, a query statement supported by a monitoring system prometheus, referred to as PromQL) of a statement type supported by the monitoring system. The monitoring system has a built-in visual management interface. Through the visual management interface, a user can easily understand the current configuration of the monitoring system and the running state of a monitoring task. Through the visual management interface, the data stored in the third-party database can be monitored. Specifically, through the visual panel of the monitoring system, the user can directly use the PromQL to query monitoring data in real time. For example, the keyword node_load1 can be used to query sample data from the third-party database storing sample data of host load. In the visual management interface, the queried sample data can be displayed in chronological order to form a trend chart of host load change over time.

[0104] PromQL is a powerful data query language customized by monitoring systems. In addition to using monitoring indicators as query keywords, PromQL also has a large number of built-in functions to help users further process time series data. For example, the rate function can calculate the change in sample data within a unit of time, i.e., the growth rate. Therefore, by using this function, the utilization rate of data processing resources can be approximately calculated by the time of resource use.

[0105] If the result of the PromQL query does not contain the target label, and the data queried by the query statement suitable for the target database carries the target label, only the without expression is used to remove the target label carried by the queried data, and then the data aggregation processing is performed to obtain the data without the target label, i.e., the query result matched with the PromQL.

[0106] In the embodiment, by aggregating the data categories as expected, the sparse data belonging to the data categories is aggregated, the data queried by the second type of query statement is processed again, the result after processing is matched with the first type of query statement, so that the database that does not support the input query statement can be queried without changing the input query statement, and the result matched with the input query statement is obtained. For computer equipment, more databases can be supported for data storage, which expands the data storage range. For users, the input method of the commonly used query statement does not need to be changed, and there is no need to develop a query interface for each database, which improves the convenience of data query.

[0107] In one embodiment, the data query method further includes a data storage process, and the data storage process includes the following steps: in response to a data collection event, performing original data collection and data preprocessing by using a target data collection method to obtain preprocessed data; performing data reporting on the preprocessed data according to a data reporting method matched with the target data collection method; and the target data stored in the target database is data written into the target database after secondary processing of the reported preprocessed data.

[0108] The original data collection refers to a process of collecting original data to be monitored. In the data collection process, the preprocessing operations such as translation, formatting and aggregation of the original data are moved forward and performed before data reporting. In the ETL (Extract-Transform-Load) operation and the first aggregation calculation of the collected data, on the one hand, the complexity of the server operation is reduced, and on the other hand, the reporting traffic and the computing resources of the server are reduced. From the perspective of use, the complexity of using the monitoring system can be simplified and the use cost can be saved. Secondary processing of the data is performed before the data is written into the database. For example, the preprocessed data is aggregated again, or the preprocessed data is filtered, dimensionally reduced, cascade calculated and multi-dimensionally calculated between indexes.

[0109] In the embodiment, by preprocessing the data before reporting, the data processing complexity of the server can be reduced, the reporting traffic can be reduced, and the computing resources of the server can be occupied. Secondary processing of the data is performed before the data is written into the database, which can improve the query performance of the database and share the storage pressure.

[0110] In one embodiment, the original data is collected by using a target data collection method, including: at least two of the following data collection methods are used to cooperatively collect data: timed collection, data push and proxy collection; wherein the data collection method corresponding to the data reporting method of the timed collection is reporting through a gateway; the data collection method corresponding to the data reporting method of the data push is reporting according to a target protocol; and the data collection method corresponding to the data reporting method of the proxy collection is reporting based on a proxy module.

[0111] The data stored in the target database can be monitoring data of the monitoring system. The timed collection refers to a process of collecting data of the monitoring system according to a specified time condition, when the current time meets the time condition. For example, the agent timed collection or the consumption kafka data collection method, the agent timed collection is mainly applied to collect system indexes or indexes of third-party components, and the telegraph is provided with a plurality of collection plug-ins. For the scene of consuming kafka data, an independent collection plug-in can be developed for the data reported by the business to realize pre-aggregation.

[0112] Data push refers to a data processing process in which a monitored object directly pushes data to a server according to a protocol based on a network connection between the monitored object and the server. For example, the push reporting of HTTP. The push reporting of HTTP supports the line-protocol protocol, and the line-protocol protocol is a data reporting protocol supported by influxdb. The line-protocol protocol supported by the monitoring system defines the index statistics method in the protocol, defines @ as the separator of the index name and the corresponding statistics method, and supports the following statistics methods: min, max, sum, first, last, and avg. The use scenario of the HTTP reporting method is end-side reporting, such as web pages, application programs, applets, and other reporting scenarios in which client-side independent existence and resource occupation are strictly required. Another type is a short-term task scenario, such as no-function services and timing tasks that are destroyed after use.

[0113] Agent collection refers to a data collection processing method in which a plug-in cooperates with an agent module, for example, an SDK combined with an agent. The agent supports the SDKs of prometheus and datadog-statsd, and the agent supports the above two types of SDK plug-ins based on telegraph development. Among them, the prometheus SDK performs aggregation on the SDK side, and persistently occupies the memory space of the business process to cache the dimension and index data. If the dimension and index data are too much, there is a risk of memory overflow. The datadog-statsd SDK performs aggregation on the agent side. When the business code calls the SDK, the SDK caches data in a queue, and sends the data to the agent after a certain amount of data is accumulated or accumulated at a certain time. Further, the agent and the business process are deployed on the same node to avoid network transmission impact.

[0114] In a specific application, as shown in Figure 7 The data collection supports the push reporting of HTTP, the combination of SDK and agent, the agent timing collection, or the system of the data collection method of consuming kafka data. The collection terminal responds to the data collection event, cooperates with the above at least two target data collection methods to collect raw data, and pre-processes the collected data to obtain pre-processed data. The pre-processed data is reported to the server according to the data reporting method matched with the target data collection method; the server performs secondary processing on the reported pre-processed data, writes the obtained data into a target database for data storage, and provides a data source for a subsequent data query process.

[0115] In the embodiment, by using at least two target data acquisition modes to cooperatively acquire raw data, the monitoring data of different monitoring objects can be synchronously acquired, the monitoring range is expanded, the data processing resources in the monitoring process can be integrated, and the resource utilization rate is improved.

[0116] In one embodiment, the secondary processing includes data aggregation processing or data multi-dimensional processing; the method further includes: determining a target data processing channel for the preprocessed data based on a data storage condition of the preprocessed data; the target data processing channel is any one of an aggregation processing channel and a multi-dimensional processing channel; and performing secondary data processing on the preprocessed data based on a data processing flow of the target data processing channel.

[0117] Specifically, after receiving the reported preprocessed data, the server needs to perform secondary data processing on the preprocessed data in order to improve query performance and share storage pressure. Two data processing channels are provided for secondary processing. One channel is used to process data reported according to a standard monitoring data model, and only secondary aggregation is performed on the data to reduce data storage. The other channel is implemented based on flink, which provides multi-dimensional processing of filtering, dimension reduction, cascade calculation, and inter-index calculation in addition to secondary aggregation of data. The preprocessed data is processed in multiple dimensions, which can reduce the data storage on one hand, and provide result data according to the business query performance requirements on the other hand to cope with high concurrency queries and anomaly detection.

[0118] In one specific application, as shown in FIG. 8, Figure 8 a data query method is provided, which specifically includes the following steps

[0119] Step 802: obtaining a first type of query statement, performing syntax analysis on the first type of query statement, and determining the syntax hierarchy architecture of the first type of query statement.

[0120] Step 804: converting the first type of query statement into a structured syntax tree according to the syntax hierarchy architecture.

[0121] Step 806: when the structured syntax tree is a single-layer syntax tree, determining a transient expression corresponding to the first type of query statement.

[0122] Step 808: when the structured syntax tree is a multi-level syntax tree, determining an expected data type of each level of the structured syntax tree.

[0123] Step 810: when the expected data type is a vector data, determining a vector expression of the level; and when the expected data type is a scalar data, determining a scalar expression of the level.

[0124] In step 812, the scalar expression and the vector expression are aggregated, and the data time range and the data label to be queried are determined.

[0125] In step 814, a range expression corresponding to the first type of query statement is constructed according to the data time range and the data label.

[0126] In step 816, a statement category suitable for a target database in which data queried by the first type of query statement is located is determined.

[0127] In step 818, a second type of query statement is constructed according to a statement construction mode corresponding to the statement category, based on the data time and the data label represented by the instantaneous expression or the interval expression.

[0128] In step 820, the second type of query statement is executed to perform data query on the target database, and target data in a target data format is obtained, the target data format being matched with the second type of query statement.

[0129] In step 822, when the expected data format of the first type of query statement is aggregated data, the target data that meets an aggregation condition is aggregated to obtain a query result matched with the first type of query statement.

[0130] In one specific embodiment, as shown in FIG. 8, before the data query process starts, a data storage process in the target database is further included, and specifically includes the following steps: Figure 9

[0131] In step 902, in response to a data collection event, at least two of a timing collection, a data push, and an agent collection are used to cooperatively collect data.

[0132] In step 904, the collected raw data is preprocessed to obtain preprocessed data.

[0133] In step 906, the preprocessed data is reported according to a data reporting mode matched with a target data collection mode.

[0134] In step 908, a target data processing channel for the preprocessed data is determined based on a data storage condition of the preprocessed data, the target data processing channel being any one of an aggregation processing channel and a multi-dimensional processing channel.

[0135] In step 910, the preprocessed data is secondarily processed based on a data processing flow of the target data processing channel, and the secondarily processed data is written into the target database.

[0136] The application further provides an application scenario of the above data query method in a monitoring system. Specifically, the application of the data query method in the monitoring system is as follows:​

[0137] As shown in Figure 10 The monitoring system includes data collection, data processing, data storage, data query and data display modules. The data collection supports http push reporting, SDK combined with agent, agent timed collection or consumption of kafka data for data collection. The terminal responds to the data collection event, cooperates with the above-mentioned at least two target data collection methods to collect raw data, and pre-processes the collected data to obtain pre-processed data. The pre-processed data is reported to the server according to the data reporting mode matched with the target data collection mode; the server performs secondary processing on the reported pre-processed data, writes the obtained data into the target database for data storage, and the data display module queries the data by sending a first type of query statement to the target database, converts the first type of query statement into a second type of query statement adapted to the target database, queries and formats the data in the target database, obtains the query result matched with the expected data format of the first type of query statement, and feeds back to the data display module for display.

[0138] Specifically, the data collection process includes raw data collection and data reporting. Raw data collection refers to the process of collecting raw data to be monitored. In the data collection process, the preprocessing operations such as translation, formatting and aggregation of raw data are moved forward and performed before data reporting. In the data collection process, the ETL operation and the first aggregation calculation are performed on the data. On the one hand, it reduces the complexity of the server operation, and on the other hand, it reduces the reporting traffic and the computing resources of the server. From the perspective of use, it can simplify the complexity of using the monitoring system and save the use cost. Before writing the data into the database, the data is processed again. For example, the pre-processed data is aggregated again, or the pre-processed data is filtered, dimensionally reduced, cascade calculated and calculated between indexes.

[0139] The data processing process includes secondary processing of the pre-processed data by the server after receiving the reported pre-processed data, to improve the query performance and share the storage pressure. Two data processing channels are provided for secondary processing. One channel is used to process data reported according to the standard monitoring data model, and only secondary aggregation is performed on the data to reduce the data storage amount. The other channel is based on flink and provides multi-dimensional processing of data filtering, dimension reduction, cascade calculation and calculation between indexes in addition to secondary aggregation of data. The pre-processed data is processed in multiple dimensions, which can reduce the data storage amount on the one hand, and provide result data according to the business query performance requirements on the other hand, to cope with high concurrency query and anomaly detection.

[0140] The data query process comprises receiving a first type of query statement sent by a query object, calculating a step length of data queried by the first type of query statement, when the step length meets a query condition, performing expression analysis on the first type of query statement to obtain a structured syntax tree, then judging whether it is a transient query based on a number of levels of the structured syntax tree, if the structured syntax tree is a single-level syntax tree, i.e. the first type of syntax data is a transient query, then directly constructing a transient expression based on the single-level syntax tree, if the structured syntax tree is a multi-level syntax tree, i.e. the first type of syntax data is not a transient query, then determining that a range expression needs to be constructed, traversing each level of the multi-level syntax tree, and the processing procedure of each level comprises: determining whether an expected data type of the level is scalar data or vector data based on a structure of the level, if it is scalar data, then constructing a scalar sub-expression or a floating-point sub-expression, if it is vector data, then constructing a vector expression, when the traversal of each level of the multi-level syntax tree is completed, a range expression corresponding to the first type of syntax data is obtained. For the range expression or the transient expression, a time range, an index and a label value corresponding to the expression are calculated, a second type of query statement compatible with a target database is constructed based on the time range, the index and the label value, data query is performed based on the second type of query statement, and query data is obtained. Since an expected data format of the first type of query statement is aggregated data, after the query data is obtained, aggregation operation is performed again to obtain aggregation results, and the aggregation results are fed back to the query object.

[0141] In one specific application, the query object is a data visualization module in a prometheus monitoring system, the data visualization module sends a first type of query statement promQL to the server in response to a user's data query request, the server calculates the step length of the data queried by the promQL, when the step length meets the query condition, expression parsing is performed on the promQL, a structured syntax tree corresponding to the promQL is obtained, and then whether it is a transient query is determined based on the number of levels of the structured syntax tree, if the structured syntax tree is a single-level syntax tree, i.e. the promQL is a transient query, a transient expression is directly constructed based on the single-level syntax tree, if the structured syntax tree is a multi-level syntax tree, i.e. the promQL is an interval query, it is determined that a range expression needs to be constructed, and each level of the multi-level syntax tree is traversed, the processing procedure of each level includes: determining whether the expected data type of the level is scalar data or vector data based on the structure of the level, if it is scalar data, a scalar sub-expression or a floating-point sub-expression is constructed, if it is vector data, a vector expression is constructed, when the traversal of each level of the multi-level syntax tree is completed, a range expression corresponding to the promQL is obtained. For the range expression or the transient expression corresponding to the promQL, the time range, the index and the label value corresponding to the expression are calculated, an influxQL compatible with the clickHouse database and the CTSDB database is constructed based on the time range, the index and the label value, the data in the clickHouse database and the CTSDB database is queried based on the influxQL, and the query result is obtained. Since the expected data format of the promQL is aggregated data, the aggregated operation is performed again after the query data is obtained, the aggregated result is obtained, and is fed back to the data visualization module.

[0142] The data storage process of the clickHouse database and the CTSDB database includes: collecting terminal data in response to a data collection event, using at least two target data collection methods of push reporting of http, combination of SDK and agent, agent timing collection or consumption of kafka data, cooperatively collecting raw data, and preprocessing the collected data to obtain preprocessed data, reporting the preprocessed data to the server in a data reporting manner matched with the target data collection method, and the server performs secondary processing on the preprocessed data after receiving the reported preprocessed data, wherein two data processing channels are provided for the secondary processing, one channel is used for processing data reported according to a standard monitoring data model, and only secondary aggregation is performed on the data to reduce the data storage amount. Another channel is realized based on flink, and is used for multi-dimensional processing such as secondary aggregation, data filtering, dimension reduction, cascade calculation and index calculation, and the secondary processed data is stored in the clickHouse database or the CTSDB database according to the data storage condition.

[0143] It is to be understood that, although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, at least some of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0144] Based on the same inventive concept, the embodiments of the present application also provide a data query device for implementing the above-mentioned data query method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data query device embodiments provided below can refer to the limitations of the data query method in the above text, which will not be repeated here.

[0145] In one embodiment, as shown in Figure 11 A data query device 1100 is provided, comprising: a statement acquisition module 1102, a statement construction module 1104, a data query module 1106, and a data conversion module 1108, wherein:

[0146] The statement acquisition module 1102 is configured to acquire a first type of query statement, determine a syntax expression and an expected data format corresponding to the first type of query statement;

[0147] The statement construction module 1104 is configured to construct a second type of query statement adapted to a target database based on the data queried by the first type of query statement according to the data query content represented by the syntax expression;

[0148] The data query module 1106 is configured to execute the second type of query statement, perform data query on the target database, and obtain target data in a target data format, wherein the target data format matches the second type of query statement;

[0149] The data conversion module 1108 is configured to perform data format conversion on the target data according to data conversion logic between the expected data format and the target data format, and obtain a query result matching the first type of query statement.

[0150] In an embodiment, the sentence obtaining module is further configured to parse the obtained first type of query sentence in syntax, and determine a syntax hierarchical architecture of the first type of query sentence; and construct a syntax expression of the first type of query sentence according to the syntax hierarchical architecture.

[0151] In an embodiment, the syntax expression comprises an instantaneous expression or a range expression; the sentence obtaining module is further configured to convert the first type of query sentence into a structured syntax tree according to the syntax hierarchical architecture; when the structured syntax tree is a single-layer syntax tree, determine an instantaneous expression corresponding to the first type of query sentence; when the structured syntax tree is a multi-layer syntax tree, traverse each layer of the structured syntax tree to obtain a range expression corresponding to the first type of query sentence.

[0152] In an embodiment, the sentence obtaining module is further configured to determine, for each layer of the structured syntax tree, a sub-expression represented by the layer; and aggregate the sub-expressions of each layer to obtain a range expression corresponding to the first type of query sentence.

[0153] In an embodiment, the sub-expression comprises a vector expression or a scalar expression; the sentence obtaining module is further configured to determine, for each layer of the structured syntax tree, an expected data type of the layer; when the expected data type is a vector data, determine a vector expression of the layer; and when the expected data type is a scalar data, determine a scalar expression of the layer.

[0154] In an embodiment, the sentence obtaining module is further configured to aggregate the sub-expressions of each layer to determine a data time range and a data label to be queried; and construct a range expression corresponding to the first type of query sentence according to the data time range and the data label.

[0155] In an embodiment, the sentence constructing module is further configured to determine a statement category adapted to a target database in which data queried by the first type of query statement is located; and construct a second type of query statement according to a statement construction manner corresponding to the statement category, based on a data time and a data label represented by the syntax expression.

[0156] In an embodiment, the data of the expected data format comprises aggregated data; and the data of the target data format comprises sparse data.

[0157] The data converting module is further configured to aggregate, according to a data category of the expected aggregated data, sparse data belonging to the data category in the sparse data to obtain a query result matched with the first type of query statement.

[0158] In one embodiment, the apparatus further comprises a data storage module, configured to, in response to a data collection event, collect original data and perform data preprocessing by using a target data collection mode to obtain preprocessed data; perform data reporting on the preprocessed data according to a data reporting mode matched with the target data collection mode; and store target data in the target database, wherein the target data is data written into the target database after secondary processing of the reported preprocessed data.

[0159] In one embodiment, the data storage module is further configured to cooperatively collect data by using at least two of a timing collection mode, a data push mode and an agent collection mode; wherein the data reporting mode corresponding to the timing collection mode is reporting through a gateway; the data reporting mode corresponding to the data push mode is reporting according to a target protocol; and the data reporting mode corresponding to the agent collection mode is reporting based on an agent module.

[0160] In one embodiment, the secondary processing includes data aggregation processing or multi-dimensional data processing; and the data storage module is further configured to determine a target data processing channel for the preprocessed data based on a data storage condition of the preprocessed data; the target data processing channel is any one of an aggregation processing channel and a multi-dimensional processing channel; and perform secondary data processing on the preprocessed data based on a data processing flow of the target data processing channel.

[0161] The data query apparatus described above can obtain a first type of query statement, determine a syntax expression and an expected data format corresponding to the first type of query statement, construct a second type of query statement that is adapted to a target database where data queried by the first type of query statement is located according to data query content represented by the syntax expression, realize conversion of the query statement, facilitate data query, perform data query on the target database by executing the second type of query statement, obtain target data having a data format matched with the second type of query statement, realize obtaining of stored data, and perform data format conversion on the target data according to data conversion logic between the expected data format and the target data format, so that the query result can be matched with the first type of query statement, and convenient and accurate query of stored data in the database is realized.

[0162] Each module in the data query apparatus described above can be realized by software, hardware and combinations thereof in whole or in part. Each module described above can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0163] In an embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in FIG. 1. Figure 12 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store monitoring data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement a data query method.

[0164] Those skilled in the art can understand that Figure 12 The structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In an embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0166] In an embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0167] In an embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0170] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0171] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data query method, characterized in that, The method includes: Obtain the first type of query statement, perform syntax parsing on the first type of query statement, and determine the syntax hierarchy structure of the first type of query statement; According to the grammatical hierarchy architecture, the first type of query statement is converted into a structured syntax tree; When the structured syntax tree is a single-layer syntax tree, the syntax expression corresponding to the first type of query statement is determined to be an instantaneous expression; When the structured syntax tree is a multi-level syntax tree, for each level of the structured syntax tree, the sub-expression represented by the corresponding level is determined. Collect sub-expressions at each level to determine the time range and data labels of the data to be queried; Based on the data time range and the data tags, construct a range expression corresponding to the first type of query statement; Determine the expected data format corresponding to the first type of query statement; Based on the target database containing the data queried by the first type of query statement, a second type of query statement adapted to the target database is constructed according to the data query content represented by the syntactic expression; The second type of query statement is executed to query the target database and obtain target data in a target data format that matches the second type of query statement. According to the data conversion logic between the expected data format and the target data format, the target data is converted to obtain query results that match the first type of query statement.

2. The method according to claim 1, characterized in that, The sub-expression includes a vector expression or a scalar expression; The step of determining the sub-expression represented by each level of the structured syntax tree includes: For each level of the structured syntax tree, determine the expected data type for that level; When the expected data type is vector data, determine the vector expression for the target level; When the expected data type is scalar data, determine the scalar expression for the target level.

3. The method according to claim 1, characterized in that, The step of constructing a second type of query statement adapted to the target database based on the data queried by the first type of query statement and according to the data query content represented by the syntactic expression includes: Based on the target database where the data queried by the first type of query statement is located, determine the statement category that is compatible with the target database; Based on the statement construction method corresponding to the statement category, and based on the data time and data labels represented by the syntactic expression, a second type of query statement is constructed.

4. The method according to claim 1, characterized in that, The desired data format includes aggregated data; the target data format includes sparse data. The step of converting the target data according to the data conversion logic between the expected data format and the target data format to obtain query results matching the first type of query statement includes: According to the data category of the data to be aggregated, the sparse data belonging to the data category are aggregated to obtain query results that match the first type of query statement.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: In response to a data acquisition event, the target data acquisition method is used to acquire raw data and preprocess the data to obtain preprocessed data. The preprocessed data is reported according to a data reporting method that matches the target data acquisition method; The target data stored in the target database is the data written into the target database after secondary processing of the reported preprocessed data.

6. The method according to claim 5, characterized in that, The raw data collection using the target data acquisition method includes: At least two of the following data collection methods—timed collection, data push, and proxy collection—are used in conjunction to collect data; Specifically, the data collection method based on timed data collection corresponds to data reporting via a gateway; the data collection method based on data push corresponds to data reporting according to a target protocol; and the data collection method based on proxy collection corresponds to data reporting based on a proxy module.

7. The method according to claim 5, characterized in that, The secondary processing includes data aggregation processing or data multidimensional processing; The method further includes: Based on the data storage conditions of the preprocessed data, a target data processing channel for the preprocessed data is determined; the target data processing channel is either an aggregation processing channel or a multidimensional processing channel. Based on the data processing flow of the target data processing channel, the preprocessed data undergoes secondary data processing.

8. A data query device, characterized in that, The device includes: The statement acquisition module is used to acquire a first type of query statement, perform syntax parsing on the first type of query statement, determine the syntax hierarchy of the first type of query statement; convert the first type of query statement into a structured syntax tree according to the syntax hierarchy; when the structured syntax tree is a single-level syntax tree, determine the syntax expression corresponding to the first type of query statement as an instantaneous expression; when the structured syntax tree is a multi-level syntax tree, for each level of the structured syntax tree, determine the sub-expression represented by the corresponding level; collect the sub-expressions of each level, determine the time range and data label of the data to be queried; construct the syntax expression corresponding to the first type of query statement as a range expression according to the data time range and the data label; and determine the expected data format corresponding to the first type of query statement. The statement construction module is used to construct a second type of query statement that is adapted to the target database based on the target database where the data queried by the first type of query statement is located, according to the data query content represented by the syntactic expression; The data query module is used to execute the second type of query statement, perform data query on the target database, and obtain target data in a target data format that matches the second type of query statement. The data conversion module is used to convert the target data according to the data conversion logic between the desired data format and the target data format, so as to obtain the query result that matches the first type of query statement.

9. The data query device according to claim 8, characterized in that, The sub-expression includes a vector expression or a scalar expression; the statement acquisition module is further configured to determine the expected data type of the target level for each level of the structured syntax tree; when the expected data type is vector data, determine the vector expression of the target level; when the expected data type is scalar data, determine the scalar expression of the target level.

10. The data query device according to claim 8, characterized in that, The statement construction module is also used to determine the statement category that is compatible with the target database based on the target database where the data queried by the first type of query statement is located; Based on the statement construction method corresponding to the statement category, and based on the data time and data labels represented by the syntactic expression, a second type of query statement is constructed.

11. The data query device according to claim 8, characterized in that, The data in the desired data format includes aggregated data; the data in the target data format includes sparse data; the data conversion module is further configured to aggregate the sparse data belonging to the data category according to the data category of the desired aggregated data, and obtain query results that match the first type of query statement.

12. The data query device according to any one of claims 8 to 11, characterized in that, The device also includes a data storage module, which, in response to a data acquisition event, performs raw data acquisition and data preprocessing using a target data acquisition method to obtain preprocessed data. The preprocessed data is reported according to a data reporting method that matches the target data acquisition method; The target data stored in the target database is the data written into the target database after secondary processing of the reported preprocessed data.

13. The data query device according to claim 12, characterized in that, The data storage module is also used to collect data collaboratively using at least two of the following data collection methods: timed collection, data push, and proxy collection; wherein, the data reporting method corresponding to the timed collection method is reporting through a gateway; the data reporting method corresponding to the data push method is reporting according to a target protocol; and the data reporting method corresponding to the proxy collection method is reporting based on a proxy module.

14. The data query device according to claim 12, characterized in that, The secondary processing includes data aggregation processing or data multidimensional processing; the data storage module is also used to determine the target data processing channel for the preprocessed data based on the data storage conditions of the preprocessed data; The target data processing channel can be either an aggregation processing channel or a multidimensional processing channel; based on the data processing flow of the target data processing channel, the preprocessed data undergoes secondary data processing.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images