Data query method and related equipment

Abstract

The embodiment of the invention provides a data query method and related equipment, and the method comprises the steps: obtaining a processing state of first electronic equipment for a data query request in response to the data query request; if the processing state of the first electronic equipment is a processable state for the data query request, analyzing the data query request to obtain configuration parameters and grammar of the data query request; determining a first OLAP engine unit corresponding to the data query request based on the configuration parameters; and sending the data query request to the first OLAP engine unit, processing the data query request based on the grammar of the data query request through the first OLAP engine unit, and obtaining a data set which is fed back by the first OLAP engine unit and corresponds to the data query request. According to the embodiment of the invention, data query can be performed through the OLAP engine based on the grammar of the data query request, so that the use cost and deployment complexity of the OLAP engine are effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data query method and related equipment. Background Art

[0002] With the rapid development of the Internet and big data technology, OLAP (Online Analytical Processing) can quickly implement data query functions by calculating multi-dimensional analytics (MDA) and is widely used in big data queries. In order to accurately respond to users' data query requests, data queries need to be performed in multiple databases, so data query services need to be connected to different OLAP engines. However, different OLAP engines use different processing syntaxes. When data query services need to deploy multiple OLAP engines, developers need to understand the syntax specifications of multiple OLAP engines, which increases the use cost and deployment complexity of OLAP engines. Summary of the invention

[0003] In view of the above, it is necessary to provide a data query method and related devices to solve the problem that when data query services need to deploy multiple OLAP engines, developers need to understand the syntax specifications of multiple OLAP engines, which increases the use cost and deployment complexity of the OLAP engines.

[0004] In a first aspect, an embodiment of the present application provides a data query method, which is applied to a first electronic device, and the method includes: in response to a data query request, obtaining a processing status of the first electronic device for the data query request; if the processing status of the first electronic device for the data query request is a processable state, parsing the data query request to obtain configuration parameters and syntax of the data query request; determining a first online analytical processing (OLAP) engine unit corresponding to the data query request based on the configuration parameters; sending the data query request to the first OLAP engine unit, and processing the data query request based on the syntax of the data query request by the first OLAP engine unit; and obtaining a data set corresponding to the data query request fed back by the first OLAP engine unit.

[0005] In a possible implementation, the configuration parameter includes an OLAP engine unit identifier, and determining the first OLAP engine unit corresponding to the data query request based on the configuration parameter includes: determining the OLAP engine unit having the OLAP engine unit identifier as the first OLAP engine unit.

[0006] In a possible implementation, the configuration parameters include data query requirements, and determining the first OLAP engine unit corresponding to the data query request based on the configuration parameters includes: determining an OLAP engine unit that meets the data query requirements as the first OLAP engine unit.

[0007] In a possible implementation, the data query request is parsed to obtain the syntax of the data query request, including: decomposing a query string corresponding to the data query request into multiple tags; and combining the multiple tags based on the syntax rules corresponding to the data query request to construct a syntax tree of the data query request.

[0008] In a possible implementation, the method further includes: verifying the syntax of the data query request based on a preset verification rule; if the syntax of the data query request passes the verification, determining the first OLAP engine unit corresponding to the data query request based on the configuration parameters.

[0009] In a possible implementation, the method further includes: if the syntax of the data query request is not compatible with the first OLAP engine unit, converting the syntax of the data query request based on the syntax rule of the first OLAP engine unit.

[0010] In a possible implementation, sending the data query request to the first OLAP engine unit includes: calling a connection processing thread corresponding to the first OLAP engine unit to distribute the data query request to the first OLAP engine unit.

[0011] In a possible implementation, the method further includes: performing sensitivity verification on the data set to determine sensitive data in the data set; desensitizing the sensitive data, and sending the desensitized data set to a user terminal corresponding to the data query request.

[0012] In a possible implementation, the method further includes: sending a data query task request corresponding to the data query request to a second electronic device, receiving a processing status of the data query request fed back by the second electronic device, and establishing a session connection with the second electronic device.

[0013] In a possible implementation, the method further includes: based on the session connection, sending the processing result of the data query task and metadata information of the processing result to the second electronic device.

[0014] In a second aspect, an embodiment of the present application provides a data query method, which is applied to a second electronic device, and the method includes: in response to a data query task request sent by a first electronic device, parsing the data query task request, and obtaining metadata information of a data query request corresponding to the data query task request; based on the metadata information, determining a processing status of the first electronic device for the data query request; sending the processing status to the first electronic device, and through the first electronic device: if the processing status is a processable status, parsing the data query request, obtaining configuration parameters and syntax of the data query request, determining a first OLAP engine unit corresponding to the data query request based on the configuration parameters, sending the data query request to the first OLAP engine unit, processing the data query request based on the syntax of the data query request by the first OLAP engine unit, and obtaining a data set corresponding to the data query request fed back by the first OLAP engine unit.

[0015] In a possible implementation, determining the processing status of the first electronic device for the data query request based on the metadata information includes: detecting the load of the data query service cluster, and if the load of the data query service cluster is less than or equal to a first preset threshold, performing an authority check on the data query request based on the metadata information; if the data query request passes the authority check, determining that the processing status of the first electronic device for the data query request is a processable state; if the load of the data query service cluster is greater than the first preset threshold, determining that the processing status of the first electronic device for the data query request is a waiting state; if the waiting time of the first electronic device for the data query request is greater than or equal to the preset waiting time threshold, determining that the processing status of the first electronic device for the data query request is a timeout state.

[0016] In a possible implementation, determining the processing status of the first electronic device for the data query request based on the metadata information also includes: if the data query request passes the permission check, determining whether the number of user accounts simultaneously accessing the first electronic device is less than or equal to a second preset threshold; if the number of user accounts simultaneously accessing the first electronic device is less than or equal to the second preset threshold, determining that the processing status of the first electronic device for the data query request is a processable state; if the number of user accounts simultaneously accessing the first electronic device is greater than the second preset threshold, determining that the processing status of the first electronic device for the data query request is the waiting state.

[0017] In a third aspect, an embodiment of the present application provides a data query device, comprising: an acquisition module, configured to obtain, in response to a data query request, a processing status of the first electronic device for the data query request; a parsing module, configured to parse the data query request if the processing status of the first electronic device for the data query request is a processable state, and obtain configuration parameters and syntax of the data query request; a determination module, configured to determine a first OLAP engine unit corresponding to the data query request based on the configuration parameters; and a query module, configured to send the data query request to the first OLAP engine unit, process the data query request by the first OLAP engine unit based on the syntax of the data query request, and obtain a data set corresponding to the data query request fed back by the first OLAP engine unit.

[0018] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device implements the above-mentioned data query method.

[0019] In a fifth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores program instructions. When the program instructions are executed on an electronic device, the electronic device implements the above-mentioned data query method.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, and when the computer program code is executed on an electronic device, the electronic device implements the above-mentioned data query method.

[0021] The data query method and related devices provided in the embodiments of the present application can receive data query requests corresponding to different OLAP engines when multiple OLAP engines are integrated in a data query service cluster, provide a unified entry and exit for multiple OLAP engines, and perform data query through the OLAP engine based on the syntax of the data query request, thereby effectively reducing the use cost and deployment complexity of the OLAP engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of an application environment of a data query method provided in an embodiment of the present application.

[0024] Figure 2 A schematic diagram of the framework of a first electronic device and a second electronic device provided in an embodiment of the present application.

[0025] Figure 3 A flowchart of a data query method provided in one embodiment of the present application.

[0026] Figure 4 A flowchart of processing a data query request provided in one embodiment of the present application.

[0027] Figure 5 A flowchart of a data query method provided in another embodiment of the present application.

[0028] Figure 6 A flowchart of a data query method provided in another embodiment of the present application.

[0029] Figure 7 A flowchart of a data query method provided in another embodiment of the present application.

[0030] Figure 8 A schematic diagram of the structure of a first data query device provided in one embodiment of the present application.

[0031] Fig. 9 A schematic diagram of the structure of a second data query device provided in one embodiment of the present application.

[0032] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] The terms "first" and "second" involved in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field in this application. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can represent A or B. "And / or" in this application is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "At least one" refers to one or more. "Multiple" refers to two or more than two. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c. In the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0036] With the rapid development of the Internet and big data technology, OLAP can quickly realize data query functions by calculating multi-dimensional analysis (MDA), and is widely used in big data queries. In order to accurately respond to users' data query requests, data queries need to be performed in multiple databases, so different OLAP engines need to be connected and different OLAP engines need to be selected in different scenarios to implement data query services. Since each OLAP engine has a unique solution scenario and SQL processing syntax, developers need to learn the syntax specifications of multiple OLAP engines before they can develop data, which increases the cost of using OLAP engines and the complexity of learning.

[0037] The direct users of the Software as a Service (SaaS) industry are tenants, and the deployment methods are divided into single-tenant cluster or multi-tenant cluster. In order to improve resource utilization, the general deployment architecture is a multi-tenant shared cluster deployment mode for resource sharing. However, the OLAP engine is a technical architecture model with high requirements for memory and CPU. A single query may schedule CPU multi-core vectorized execution. Therefore, in the scenario of multi-tenant requests, there is concurrent resource preemption, which affects each other. At present, the resource isolation of mainstream OLAP engines such as StarRocks and Clickhouse is account-level virtual isolation, not physical isolation, and there are still scenarios of resource preemption, which will affect the business of other tenants.

[0038] Business access to data query is directly connected to the OLAP engine through reporting tools and business intelligence BI tools, and does not provide edge functions such as global current limiting, circuit breaking, downgrade, SQL check rewrite, and caching. OLAP queries are vectorized queries. If there are special situations such as excessive access volume and a large number of SQL queries, the OLAP engine may fail (OLAP queries are vectorized queries).

[0039] In addition, clusters can be deployed independently for different businesses to isolate the impact of different businesses, but the utilization rates of each business are different, resulting in a large amount of resource waste in different clusters during idle periods. For example, the utilization rate of the OLAP engine for commercial reports is relatively average, with a certain amount of queries every day, while the utilization rate of the OLAP engine for finance is concentrated at the end of the month when financial statistics are performed, so the overall cluster resources are used less during normal times and more during special periods.

[0040] In order to solve the problem that when a data query service needs to deploy multiple OLAP engines, developers need to understand the syntax specifications of multiple OLAP engines, which increases the use cost and deployment complexity of the OLAP engine, the embodiment of the present application provides a data query method, which can integrate multiple OLAP engines in a data query service cluster, receive data query requests corresponding to different OLAP engines sent by user terminals, provide a unified entrance and exit for multiple OLAP engines, and process the data query request through the corresponding OLAP engine based on the syntax of the data query request to obtain the queried data set, effectively reducing the use cost and deployment complexity of the OLAP engine. In addition, the data query method provided in the embodiment of the present application also supports high-concurrency and low-latency online data query services, flexibly controls the scheduling and use of permissions and resources, and thus refines resource management.

[0041] See also Figure 1 , which is a schematic diagram of an application environment of a data query method provided in an embodiment of the present application. Figure 1 The data query system 100 shown includes a first electronic device 10, a second electronic device 20, a gateway 30, and a user terminal 40 connected via network communication. The number of the first electronic device 10 can be one or more. The first electronic device 10, the second electronic device 20, and the gateway 30 constitute a data query service cluster or a data query service system for providing data query services.

[0042] In some embodiments of the present application, the first electronic device 10 and the second electronic device 20 may be a single server, a cloud server or a server cluster, the gateway 30 may be a router, and the user terminal 40 may be a smart phone, a personal computer, a smart wearable device, etc. The first electronic device 10 and the second electronic device 20 may deploy a computer program product (e.g., software code, computer-readable instructions, etc.) programmed and implemented according to the data query method provided in the embodiment of the present application, thereby providing a data query service. The first electronic device 10 is the execution end of the data query service, and the second electronic device 20 is the management end of the data query service.

[0043] See also Figure 2 As shown, it is a schematic diagram of the framework of the first electronic device and the second electronic device provided by an embodiment of the present application. The first electronic device (MixDB Proxy) includes a communication layer of the MySQL protocol, an application connection pool, an SQL parsing component, an SQL optimization component, an SQL routing component, a process control component, an SQL execution component, a database connection pool, a JDBC driver, and an HTTP CLIENT. It can also provide multiple auxiliary functions, such as authentication, metadata parsing, routing strategy, session management, heartbeat detection, memory management, thread management, etc. The first electronic device is connected to multiple OLAP engines through a JDBC driver or HTTP CLIENT. Multiple OLAP engines are applied to one or more databases. Multiple OLAP engines may include StarRocks, Clickhouse, ES, and KV. The user terminal can access the resources of the first electronic device through an application to realize data query. The first electronic device can synchronize configuration and status with the second electronic device.

[0044] The second electronic device (MixDB Manager) includes a network (web) service module, a core (core) service module and a dependent (depend) service module. The network service module is used to provide a work order center, data exploration, and engine management. The core service module is used to provide engine management, connection pool management, library management, table management, column management, partition management, account management, and hit (query result) management. The dependent service module is used to provide Dmall ID (user identifier), configuration center, and process engine.

[0045] See also Figure 3 The figure is a flow chart of a data query method provided by an embodiment of the present application. The method is applied to Figure 1 and Figure 2 In the first electronic device shown, the data query method includes: S101, in response to a data query request, obtaining a processing status of a first electronic device for the data query request.

[0046] In one embodiment of the present application, the user terminal sends a data query request to the gateway, and the gateway sends the data query request to an idle first electronic device based on a load balancing strategy. The idle first electronic device may be a first electronic device whose CPU usage is less than or equal to a preset threshold (e.g., 20%), and the first electronic device is the execution end of the data query service. After receiving the data query request, the first electronic device creates a data query task in response to the data query request, sends a data query task request to the second electronic device, requests to create a session connection with the second electronic device, and receives the processing status fed back by the second electronic device based on the data query task request, which is the processing status of the first electronic device for the data query request. Among them, the processing status includes but is not limited to: a processable state, a waiting state, a timeout state, and an unprocessable state. The processable state means that the first electronic device can currently process the data query request. The waiting state means that the first electronic device needs to wait for a preset waiting time before it can process the data query request. The timeout state means that the time the first electronic device waits to process the data query request has exceeded the preset waiting time. If the processing state is the timeout state, the first electronic device may not process the data query request, or may process the data query request when the processing state changes to a processable state (for example, when the load of the data query service cluster is less than or equal to the first preset threshold). The unprocessable state means that the first electronic device is currently unable to process the data query request.

[0047] S102: If the processing status of the first electronic device is a processable status for the data query request, the data query request is parsed to obtain configuration parameters and syntax of the data query request.

[0048] In one embodiment of the present application, if the processing status of the data query request is a processable status, it is determined that the data query service cluster (i.e., the data query service system) can currently process the data query request, a data query task execution thread is created, the data query request is parsed, and the configuration parameters of the data query request are obtained. The configuration parameters may be an OLAP engine unit identifier or a data query requirement.

[0049] In one embodiment of the present application, a data query request is parsed to obtain the syntax of the data query request, including: decomposing the query string corresponding to the data query request into multiple tokens, where a token is the smallest grammatical unit of the query language, and may be a keyword, an identifier, a literal, an operator, etc., and combining multiple tokens based on the grammatical rules corresponding to the data query request to construct a syntax tree for the data query request. For example, the query string of the data query request is: SELECT name, age FROM users WHERE age>30, and the multiple tokens obtained by decomposition include SELECT, name, age, FROM, users, WHERE, age,>, 30. This query string is an SQL query statement, and an abstract syntax tree is constructed based on the grammatical rules of SQL. Each node of the abstract syntax tree is a component of the query, such as a select list, a FROM clause, a WHERE clause, etc. Thus, the syntax tree constructed based on the query string may be: SELECT / \ COLUMN_LIST FROM / \ / name age users / WHERE / age>30.

[0050] S103: Determine a first OLAP engine unit corresponding to the data query request based on the configuration parameters.

[0051] In one embodiment of the present application, the configuration parameter is an OLAP engine unit identifier, and each OLAP engine unit connected to the first electronic device has a corresponding OLAP engine unit identifier. After parsing and obtaining the OLAP engine unit identifier in the data query request, the OLAP engine unit identifier in the data query request is compared with the OLAP engine unit identifier of each OLAP engine unit to determine whether the OLAP engine unit identifier in the data query request is the same as the OLAP engine unit identifier of any OLAP engine unit. If the OLAP engine unit identifier in the data query request is the same as the OLAP engine unit identifier of an OLAP engine unit, the OLAP engine unit having the OLAP engine unit identifier in the data query request is determined as the first OLAP engine unit, and the first OLAP engine unit is the OLAP engine unit used to execute the data query request.

[0052] In another embodiment of the present application, the configuration parameter is a data query requirement, for example, the data query requirement includes data selection, data filtering, and data aggregation, and each OLAP engine unit connected to the first electronic device has a corresponding data query function, and the data query function includes data selection, data filtering, and data aggregation. After parsing the data query requirement in the data query request, the data query requirement is compared with the data query function of each OLAP engine unit to determine whether the data query requirement is the same as the data query function of any OLAP engine unit. If the data query requirement is the same as the data query function of any OLAP engine unit, the OLAP engine unit is determined to meet the data query requirement, so that the OLAP engine unit with the same data query function as the data query requirement is determined as the first OLAP engine unit.

[0053] S104, sending the data query request to the first OLAP engine unit, processing the data query request based on the syntax of the data query request by the first OLAP engine unit, and obtaining a data set corresponding to the data query request fed back by the first OLAP engine unit.

[0054] In one embodiment of the present application, after determining the first OLAP engine unit for processing the data query request, the data query request is sent to the first OLAP engine unit. If the syntax of the data query request is compatible with the first OLAP engine unit, the data query request is directly processed by the first OLAP engine unit to obtain the data set corresponding to the data query request. If the syntax of the data query request is not compatible with the first OLAP engine unit, the syntax of the data query request is converted based on the syntax rules of the first OLAP engine unit so that the syntax of the data query request is compatible with the first OLAP engine unit, and then the data query request is processed by the first OLAP engine unit to obtain the data set corresponding to the data query request.

[0055] Through the above-mentioned embodiments of the present application, when multiple OLAP engines are integrated in a data query service cluster, data query requests corresponding to different OLAP engines can be received, a unified entrance and exit can be provided for multiple OLAP engines, and data queries can be performed through the OLAP engine based on the syntax of the data query request, thereby effectively reducing the use cost and deployment complexity of the OLAP engine.

[0056] See also Figure 4 As shown, it is a flowchart of processing data query request provided by an embodiment of the present application.

[0057] S1041: Send a data query request to the first OLAP engine unit.

[0058] In one embodiment of the present application, each OLAP engine unit has a corresponding connection processing thread, which is used to establish a connection with the first electronic device and obtain a data query request sent by the first electronic device. The first electronic device obtains and calls the connection processing thread corresponding to the first OLAP engine unit and distributes the data query request to the first OLAP engine unit.

[0059] S1042, determine whether the syntax of the data query request is compatible with the first OLAP engine unit. If the syntax of the data query request is compatible with the first OLAP engine unit, the process proceeds to S1043; if the syntax of the data query request is not compatible with the first OLAP engine unit, the process proceeds to S1044.

[0060] In one embodiment of the present application, it is determined whether the syntax of each operation in the query statement of the data query request is compatible with the syntax rules of the first OLAP engine unit. If the syntax of each operation in the query statement of the data query request matches the syntax rules of the first OLAP engine unit, it is determined that the syntax of the data query request is compatible with the first OLAP engine unit. If the syntax of any operation in the query statement of the data query request does not match the syntax rules of the first OLAP engine unit, it is determined that the syntax of the data query request is incompatible with the first OLAP engine unit. For example, although the OLAP engine unit Clickhouse supports SQL data query requests, the join syntax corresponding to the connection operation of Clickhouse is inconsistent with the standard SQL. In this way, if the first OLAP engine unit is Clickhouse and the data query request is standard SQL, it is determined that the syntax of the connection operation in the query statement of the data query request is incompatible with the syntax rules of the first OLAP engine unit.

[0061] S1043: Process the data query request through the first OLAP engine unit to obtain a data set corresponding to the data query request.

[0062] In one embodiment of the present application, after receiving a data query request, the first OLAP engine unit generates an execution plan based on a syntax tree of the data query request, retrieves data in a database according to the execution plan, obtains a data set, and sends the data set to the first electronic device.

[0063] S1044: Convert the syntax of the data query request based on the syntax rule of the first OLAP engine unit.

[0064] In one embodiment of the present application, if the syntax of the data query request is not compatible with the first OLAP engine unit, the syntax of the data query request is converted based on the syntax rules of the first OLAP engine unit to make the syntax of the data query request compatible with the first OLAP engine unit. Specifically, if the syntax of the data query request is not compatible with the first OLAP engine unit, the operation whose syntax does not match the syntax rules of the first OLAP engine unit is determined, and the syntax corresponding to the operation is converted into the syntax corresponding to the operation in the syntax rules of the first OLAP engine unit. For example, the data query request of the standard SQL can be: SELECT a.*, b.* FROM table_a a JOIN table_b b ON a.id = b.id.

[0065] The syntax of the join operation is JOIN. If the first OLAP engine unit is Clickhouse, the syntax of the join operation in the syntax rule of Clickhouse is ANY LEFT JOIN. In this case, the syntax of the join operation in the data query request needs to be converted, that is, JOIN is changed to ANY LEFT JOIN. The converted data query request can be: SELECT a.*, b.* FROM table_a a ANY LEFT JOIN table_b b ON a.id.

[0066] In another embodiment of the present application, the determination of whether the syntax of the data query request is compatible with the first OLAP engine unit and the syntax conversion of the data query request may also be performed before the data query request is sent to the first OLAP engine unit.

[0067] Through the above embodiments of the present application, when the syntax of the data query request is not compatible with the OLAP engine, the syntax of the data query request can be converted, which effectively reduces the use cost and deployment complexity of the OLAP engine.

[0068] See also Figure 5 FIG. 1 is a flowchart of a data query method provided by another embodiment of the present application. The method is applied to Figure 1 and Figure 2 In the first electronic device shown, the data query method includes: S201: In response to a data query request, obtain a processing status of a first electronic device for the data query request.

[0069] S202: If the processing status of the first electronic device is a processable status for the data query request, the data query request is parsed to obtain configuration parameters and syntax of the data query request.

[0070] S203, based on the preset verification rules, the syntax of the data query request is verified to determine whether the syntax of the data query request passes the verification. If the syntax of the data query request passes the verification, the process enters S204; if the syntax of the data query request fails the verification, the process enters S206.

[0071] In one embodiment of the present application, the preset verification rules include, but are not limited to: syntax checking, injection detection, resource usage restrictions, data integrity constraints, predefined rules, and performance considerations. Among them, syntax checking is used to verify whether the syntax of the data query request complies with the syntax rules, injection detection is used to identify whether the data query request contains malicious input, resource usage restrictions are used to detect whether the data query request needs to consume too many system resources, data integrity constraints are used to detect whether the data query request will destroy the data integrity of the database, predefined rules are used to detect whether the data query request violates the predefined rules, for example, the predefined rules prohibit the deletion or modification of data, and performance considerations are used to detect whether the data query request affects the system performance. If the data query request passes the verification of all the above verification rules, it is determined that the data query request complies with the specifications. If the data query request fails to pass the verification of any verification rule, it is determined that the data query request does not comply with the specifications.

[0072] S204: Determine a first OLAP engine unit corresponding to the data query request based on the configuration parameters.

[0073] S205: Send the data query request to the first OLAP engine unit, and the first OLAP engine unit processes the data query request based on the syntax of the data query request to obtain a data set corresponding to the data query request fed back by the first OLAP engine unit.

[0074] S206, feedback abnormal information.

[0075] In one embodiment of the present application, the first electronic device sends the abnormal information to the user terminal corresponding to the data query request through the gateway to feedback the abnormal information. The abnormal information is used to indicate that the data query request processing fails.

[0076] The specific implementations of S201-S202 and S204-S205 are the same as those of S101-S104 and will not be elaborated here.

[0077] Through the above-mentioned embodiments of the present application, the syntax of the data query request can be verified to ensure that the syntax of the data query request complies with the verification rules, to ensure that the data query request can be processed normally, and to effectively improve the efficiency of the data query request.

[0078] See also Figure 6 FIG. 1 is a flowchart of a data query method provided by another embodiment of the present application. The method is applied to Figure 1 and Figure 2 In the first electronic device shown, the data query method includes: S401: In response to a data query request, obtain a processing status of a first electronic device for the data query request.

[0079] S402: If the processing status of the first electronic device is a processable status for the data query request, the data query request is parsed to obtain configuration parameters and syntax of the data query request.

[0080] S403, based on the preset verification rules, the syntax of the data query request is verified to determine whether the syntax of the data query request passes the verification. If the syntax of the data query request passes the verification, the process enters S404; if the syntax of the data query request fails the verification, the process enters S409.

[0081] S404: Determine a first OLAP engine unit corresponding to the data query request based on the configuration parameters.

[0082] S405: Send the data query request to the first OLAP engine unit, and the first OLAP engine unit processes the data query request based on the syntax of the data query request to obtain a data set corresponding to the data query request fed back by the first OLAP engine unit.

[0083] S406, based on the data sensitivity rules, perform sensitivity check on the data set to determine whether the data set contains sensitive data. If the data set contains sensitive data, the process proceeds to S407; if the data set does not contain sensitive data, the process proceeds to S408.

[0084] In one embodiment of the present application, the data sensitivity rule includes: determining data of a preset type as sensitive data, the preset types include but are not limited to: personal identity information, financial data, medical records, passwords. Determine whether each data in the data set is data of the preset type, and if any data is data of the preset type, determine the data as sensitive data.

[0085] S407, desensitizing the sensitive data.

[0086] In one embodiment of the present application, desensitization processing includes but is not limited to: data masking, data replacement, data generalization, data encryption, data randomization, and data access control. Data masking refers to hiding sensitive data. Data replacement refers to replacing sensitive data with non-sensitive fictitious data while keeping the data format unchanged. Data generalization refers to replacing specific data with broad categories to reduce the recognition of the data. Data encryption refers to encrypting sensitive data, such as hash algorithm encryption. Data randomization refers to converting sensitive data into irreversible random data. Data access control refers to setting access rights for sensitive data to ensure that only authorized users can access it.

[0087] S408, feedback data set.

[0088] In one embodiment of the present application, the first electronic device sends the data set to the user terminal corresponding to the data query request through the gateway to feedback the data set.

[0089] S409, feedback abnormal information.

[0090] The specific implementations of S401-S405 and S409 are the same as those of S201-S206, and are not elaborated here.

[0091] Through the above-mentioned embodiments of the present application, after the data set corresponding to the data query request is queried, the sensitive data in the data set is desensitized to effectively ensure data security.

[0092] See also Figure 7 FIG. 1 is a flowchart of a data query method provided by another embodiment of the present application. The method is applied to Figure 1 and Figure 2 In the second electronic device shown, the data query method includes: S501: Receive a data query task request sent by a first electronic device in response to a data query request.

[0093] In one embodiment of the present application, the data query task request includes a connection request, a resource request, and a data query request sent by a user terminal.

[0094] S502, parsing the data query task request to obtain metadata information of the data query request corresponding to the data query task request.

[0095] S503: Determine a processing status of the first electronic device for the data query request based on the metadata information.

[0096] In one embodiment of the present application, the second electronic device is a management terminal of a data query service, which is used to manage the execution of data query requests by one or more first electronic devices. The second electronic device receives a data query task request sent by the first electronic device, responds to a connection request in the data query task request, establishes a session connection with the first electronic device, responds to a resource request in the data query task request, allocates processing resources to the first electronic device, for example, the processing resources may be virtual machines, storage resources, etc., and creates a context for the data query task request. Context refers to the relevant information and settings required when executing a data query. Among them, the context of the data query task request includes but is not limited to: user information, session information, query parameters, time information, location information, and transaction information.

[0097] In one embodiment of the present application, the second electronic device creates a transaction file based on the data query task request, and stores the context in a directory of the transaction file snapshot.

[0098] In one embodiment of the present application, the second electronic device performs a permission check on the data query request to determine whether the user who sends the data query request has the data query permission. If the user who sends the data query request has the data query permission, it is determined that the processing state of the first electronic device is a processable state for the data query request; if the user who sends the data query request does not have the data query permission, it is determined that the processing state of the first electronic device is an unprocessable state for the data query request. The second electronic device parses the data query request, obtains metadata information in the data query request, and the metadata information includes library metadata information, table metadata information, column metadata information, and partition metadata information, and calls the metadata interface to perform permission check on the data query request. For example, it is determined whether the user account that sends the data query request is a registered account. If the user account is a registered account, it is determined that the user has passed the identity authentication, and based on the user identity, the user's access rights to the requested database, table, column, and partition are checked. For example, the metadata information of the database and the data table is mapped to the permission system to check whether the user has the permission to access the database, table, column, and partition.

[0099] In another embodiment of the present application, the second electronic device detects the load of the data query service cluster and determines the processing status of the data query request. If the load of the data query service cluster is less than or equal to the preset threshold, it is determined that the processing status of the first electronic device is a processable state for the data query request. If the load of the data query service cluster is greater than the preset threshold, it is determined that the processing status of the first electronic device is a waiting state for the data query request. Among them, the load of the data query service cluster can be the number of data query requests currently being processed, in which case the preset threshold can be 100, 200, 300 or other values. In other embodiments of the present application, the load of the data query service cluster can also be the average CPU usage of multiple second electronic devices, in which case the preset threshold can be 85%, 90%, 95% or other percentage values.

[0100] In another embodiment of the present application, if the waiting time of the data query request is greater than or equal to a preset waiting time threshold, it is determined that the processing state of the first electronic device for the data query request is a timeout state. For example, the preset waiting time threshold may be 1 minute, 2 minutes, 5 minutes or other time.

[0101] In another embodiment of the present application, if the load of the data query service cluster is less than or equal to a preset threshold, the second electronic device performs an authority check on the data query request to determine whether the user sending the data query request has data query authority. If the user sending the data query request has data query authority, it is determined that the processing status of the first electronic device for the data query request is a processable state; if the user sending the data query request does not have data query authority, it is determined that the processing status of the first electronic device for the data query request is an unprocessable state.

[0102] In another embodiment of the present application, if the data query request passes the authority check, the second electronic device performs flow control detection of the user account, determines whether the number of user accounts accessing the first electronic device at the same time is less than or equal to the preset flow control threshold, and if the number of user accounts accessing the first electronic device at the same time is less than or equal to the preset flow control threshold, determines that the processing state of the first electronic device for the data query request is a processable state; if the number of user accounts accessing the first electronic device at the same time is greater than the preset flow control threshold, the user account corresponding to the data query request is added to the request queue pool, and the processing state of the first electronic device for the data query request is updated to a waiting state. For example, the preset flow control threshold is 100, 200 or other values.

[0103] In one embodiment of the present application, after the second electronic device sends the processing status to the first electronic device, it also stores the processing status in the transaction log of the transaction file.

[0104] S504, sending the processing status to the first electronic device. If the processing status is a processable state, the first electronic device parses the data query request, obtains the configuration parameters and syntax of the data query request, determines the first OLAP engine unit corresponding to the data query request based on the configuration parameters, sends the data query request to the first OLAP engine unit, processes the data query request based on the syntax of the data query request by the first OLAP engine unit, and obtains the data set corresponding to the data query request fed back by the first OLAP engine unit.

[0105] In one embodiment of the present application, the first electronic device also sends the processing result of the data query request and metadata information of the processing result to the second electronic device based on the session connection with the second electronic device.

[0106] In one embodiment of the present application, after the second electronic device receives the processing result of the data query request and the metadata information of the processing result, it updates the context and transaction log of the data query request, updates the processing status of the data query request to completed, closes the transaction corresponding to the data query request, and ends the processing of the data query request.

[0107] Through the above-mentioned embodiments of the present application, the management end of the data query service cluster can determine the processing status of the data query request based on the cluster load and the metadata information of the data query request, while ensuring the system performance, and at the same time ensuring that the data query request can be processed normally, thereby effectively improving the processing efficiency of the data query request.

[0108] See also Figure 8 As shown, it is a structural diagram of a first data query device provided by an embodiment of the present application. The first data query device 200 may include a plurality of functional modules composed of computer program segments. The computer program of each program segment in the first data query device 200 may be stored in a memory of the first electronic device and executed by at least one processor to perform the data query function.

[0109] In one embodiment of the present application, the first data query device 200 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an acquisition module 201, a parsing module 202, a determination module 203 and a query module 204. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In one embodiment of the present application, the first data query device 200 can be used to implement the data query method provided in the embodiment of the present application.

[0110] The acquisition module 201 is used for acquiring the processing status of the first electronic device in response to the data query request.

[0111] The parsing module 202 is used to parse the data query request and obtain the configuration parameters and syntax of the data query request if the processing status of the first electronic device is a processable status for the data query request.

[0112] The determination module 203 is used to determine the first OLAP engine unit corresponding to the data query request based on the configuration parameters.

[0113] The query module 204 is used to send the data query request to the first OLAP engine unit, and the first OLAP engine unit processes the data query request based on the syntax of the data query request to obtain a data set corresponding to the data query request fed back by the first OLAP engine unit.

[0114] See also Fig. 9 As shown, it is a structural diagram of a second data query device provided by an embodiment of the present application. The second data query device 300 may include a plurality of functional modules composed of computer program segments. The computer program of each program segment in the second data query device 300 may be stored in a memory of the second electronic device and executed by at least one processor to perform the data query function.

[0115] In one embodiment of the present application, the second data query device 300 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: a receiving module 301, a parsing module 302, and a sending module 303. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In one embodiment of the present application, the second data query device 300 can be used to implement the data query method provided in the embodiment of the present application.

[0116] The receiving module 301 is used to receive a data query task request sent by the first electronic device in response to a data query request.

[0117] The parsing module 302 is used to respond to the data query task request sent by the first electronic device, parse the data query task request, obtain metadata information of the data query request corresponding to the data query task request, and determine the processing status of the first electronic device for the data query request based on the metadata information.

[0118] The sending module 303 is used to send the processing status to the first electronic device.

[0119] See also Fig.10As shown, it is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. Electronic device 1 can be a server. For example, electronic device 1 can be a central server, an edge server, or a local server of a local data center. Electronic device 1 can be the first electronic device in the above-mentioned embodiment of the present application, or it can be the second electronic device in the above-mentioned embodiment of the present application. Electronic device 1 may include a communication module 11, a memory 12, a processor 13, an input / output (I / O) interface 14 and a bus 15. The processor 13 is respectively coupled to the communication interface 11, the memory 12, and the I / O interface 14 through the bus 15.

[0120] The communication module 11 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), bluetooth (bluetooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0121] The memory 12 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 13, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0122] The non-volatile memory may also store executable programs and user and application data, etc., and may be pre-loaded into the random access memory for direct reading and writing by the processor 13. The non-volatile memory may include a disk storage device and a flash memory.

[0123] The memory 12 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 13. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 13, the data query method in the above embodiment of the present application can be implemented on the electronic device 1.

[0124] In other embodiments, the computer device 1 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the computer device 1 .

[0125] The processor 13 may include one or more processing units, for example, the processor 13 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0126] The processor 13 provides computing and control capabilities. For example, the processor 13 is used to execute computer programs stored in the memory 12 .

[0127] The I / O interface 14 is used to provide a channel for user input or output. For example, the I / O interface 14 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0128] The bus 15 is at least used to provide a channel for mutual communication among the communication module 11 , the memory 12 , the processor 13 , and the I / O interface 14 in the computer device 1 .

[0129] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1. In other embodiments of the present application, the electronic device 1 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0130] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0131] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0132] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] In the embodiments provided in the present application, it should be understood that the disclosed device / user terminal and method can be implemented in other ways. For example, the device / user terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data query method, characterized in that: Applied to a first electronic device, the method includes: In response to a data query request, obtaining a processing status of the first electronic device for the data query request; If the processing state of the first electronic device is a processable state for the data query request, parsing the data query request to obtain configuration parameters and syntax of the data query request; Determine a first online analytical processing (OLAP) engine unit corresponding to the data query request based on the configuration parameters; Sending the data query request to the first OLAP engine unit, and processing the data query request by the first OLAP engine unit based on the syntax of the data query request; A data set corresponding to the data query request fed back by the first OLAP engine unit is obtained.

2. The data query method according to claim 1, characterized in that: The configuration parameter includes an OLAP engine unit identifier, and determining the first OLAP engine unit corresponding to the data query request based on the configuration parameter includes: An OLAP engine unit having the OLAP engine unit identifier is determined as the first OLAP engine unit.

3. The data query method according to claim 1, characterized in that: The configuration parameters include data query requirements, and the determining the first OLAP engine unit corresponding to the data query request based on the configuration parameters includes: An OLAP engine unit that meets the data query requirement is determined as the first OLAP engine unit.

4. The data query method according to claim 1, characterized in that: Parsing the data query request to obtain the syntax of the data query request includes: Decomposing a query string corresponding to the data query request into multiple tags; Based on the grammatical rules corresponding to the data query request, the multiple tags are combined to construct a grammatical tree of the data query request.

5. The data query method according to claim 4, characterized in that: The method further comprises: Verify the syntax of the data query request based on preset verification rules; If the syntax of the data query request passes the verification, a first OLAP engine unit corresponding to the data query request is determined based on the configuration parameters.

6. The data query method according to claim 1 or 4, characterized in that: The method further comprises: If the syntax of the data query request is not compatible with the first OLAP engine unit, the syntax of the data query request is converted based on the syntax rules of the first OLAP engine unit.

7. The data query method according to claim 1, characterized in that: The sending the data query request to the first OLAP engine unit includes: The connection processing thread corresponding to the first OLAP engine unit is called to distribute the data query request to the first OLAP engine unit.

8. The data query method according to claim 1, characterized in that: The method further comprises: Performing sensitivity verification on the data set to determine sensitive data in the data set; The sensitive data is desensitized, and the desensitized data set is sent to the user terminal corresponding to the data query request.

9. The data query method according to claim 1, characterized in that: The method further comprises: A data query task request corresponding to the data query request is sent to a second electronic device, a processing status of the data query request fed back by the second electronic device is received, and a session connection is established with the second electronic device.

10. The data query method according to claim 9, characterized in that: The method further comprises: Based on the session connection, the processing result of the data query task and the metadata information of the processing result are sent to the second electronic device.

11. A data query method, characterized in that: Applied to a second electronic device, the method includes: In response to the data query task request sent by the first electronic device, parsing the data query task request to obtain metadata information of the data query request corresponding to the data query task request; Determining, based on the metadata information, a processing status of the first electronic device for the data query request; The processing status is sent to the first electronic device, and through the first electronic device: if the processing status is a processable state, the data query request is parsed to obtain configuration parameters and syntax of the data query request, and a first OLAP engine unit corresponding to the data query request is determined based on the configuration parameters, and the data query request is sent to the first OLAP engine unit. The data query request is processed by the first OLAP engine unit based on the syntax of the data query request, and a data set corresponding to the data query request fed back by the first OLAP engine unit is obtained.

12. The data query method according to claim 11, characterized in that: The determining, based on the metadata information, a processing status of the first electronic device for the data query request includes: Detecting a load of a data query service cluster, and if the load of the data query service cluster is less than or equal to a first preset threshold, performing an authority check on the data query request based on the metadata information; If the data query request passes the authority check, determining that the processing status of the first electronic device for the data query request is a processable status; If the load of the data query service cluster is greater than the first preset threshold, determining that the processing state of the first electronic device for the data query request is a waiting state; If the waiting time of the first electronic device for the data query request is greater than or equal to a preset waiting time threshold, it is determined that the processing status of the first electronic device for the data query request is a timeout status.

13. An electronic device, characterized in that: The electronic device comprises a memory and a processor: Wherein, the memory is used to store program instructions; The processor is used to read and execute the program instructions stored in the memory. When the program instructions are executed by the processor, the electronic device implements the data query method as described in any one of claims 1 to 10, or implements the data query method as described in any one of claims 11 to 12.

14. A computer storage medium, characterized in that: The computer storage medium stores program instructions, and when the program instructions are executed on an electronic device, the electronic device implements the data query method as claimed in any one of claims 1 to 10, or implements the data query method as claimed in any one of claims 11 to 12.

Images