A Druid database data query method, device and computer equipment
By obtaining the health status of each database and generating query priority, selecting the database with the best health status for querying, the problem of high failure rate and poor quality of data query results in cross-campus Druid database deployment is solved, and query accuracy and success is improved rate, enhance high availability.
Patent Information
- Application Number
- CN202110350307.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-03-31
AI Technical Summary
The existing technology of Druid database deployment in the mid-to-span campus has problems with high failure rate and poor quality of data query results, especially the waste of network bandwidth and increased pressure on management nodes during the data ingestion phase, and the high availability caused by the dependency module not supporting cross-span campus deployment. Inadequate ability.
By receiving user data query requests, the health status of each database is obtained, and query priority is generated based on the node health status, data health status, business processing volume health status and hardware health status, and the database with the best health status is selected for data query. Ensure the accuracy and success rate of query results.
It effectively improves the accuracy and success rate of data queries, avoids jitter in query results, reduces the pressure on management nodes, and enhances the high availability of Druid database.
Smart Images

Figure CN112905709B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database technology, and in particular to a Druid database data query method, device and computer equipment. Background Art
[0002] Druid is a distributed database that is widely used because it can provide data query services with extremely large data scale and millisecond latency, as well as its excellent performance in data query with high real-time requirements.
[0003] Druid is usually deployed in a single park (also called a data center). However, cross-park deployment can greatly improve Druid's high availability. Therefore, cross-park Druid deployment has gradually attracted people's attention. In the prior art, cross-park Druid deployment usually adopts the following methods: 1. All Druid nodes are distributed and deployed in multiple parks; 2. Druid clusters are deployed in two parks respectively, and then two Druid clusters are proxied by proxy devices. However, both deployment methods have some defects. Specifically, in method 1, during the data ingestion stage, Druid works in multiple nodes, and a merge operation is required before writing to deep storage, which seriously wastes cross-park network bandwidth, takes longer time, further increases the pressure on management nodes, and easily causes a vicious cycle that leads to the downtime of the entire cluster. In addition, Druid's dependent modules, such as HDFS (Hadoop Distributed File System), do not support cross-park deployment. If these modules go down, the entire Druid cluster will go down, making Druid's high availability insufficient. In method 2, since the proxy device can only detect whether the Druid cluster is down, if some nodes are offline, it will still assume that Druid can work normally, and then poll the Druid cluster to complete the data query business. At this time, although the Druid cluster is not down, data may be sometimes available and sometimes not, resulting in poor query results.
[0004] In view of the above-mentioned defects in the prior art, this paper aims to provide a Druid database data query method, device and computer storage configuration, which can effectively improve the data query accuracy and data query success rate. Summary of the invention
[0005] In view of the above problems in the prior art, the purpose of this article is to provide a Druid database data query method, device and computer equipment to solve the defects of high failure rate and poor quality of data query results in the prior art.
[0006] In order to solve the above technical problems, the specific technical solutions of this article are as follows:
[0007] On the one hand, this article provides a Druid database data query method, including:
[0008] Receive data query requests from users;
[0009] According to the data query request, obtaining health status of multiple databases, wherein the databases are respectively deployed in different parks;
[0010] generating a query priority order according to the health status of each of the databases;
[0011] According to the query priority order, select the database with the best health status to perform data query to obtain the query result;
[0012] Feedback the query results to the user.
[0013] Specifically, obtaining the health status of multiple databases according to the query requirement information includes:
[0014] Parsing the data query request to obtain a first data storage unit required by the data query request;
[0015] Counting the first dimension cardinality information of the first data storage unit corresponding to each database;
[0016] The data health status of each database is obtained according to the first dimension cardinality information of each database.
[0017] Preferably, the obtaining of the health status of the plurality of databases further comprises:
[0018] Acquire the second data storage unit of each of the databases according to a preset screening rule;
[0019] Counting the second dimension cardinality information of the second data storage unit corresponding to each database;
[0020] The data health status of each database is obtained according to the second dimension cardinality information of each database.
[0021] Specifically, obtaining the health status of the plurality of databases according to the data query request further includes:
[0022] According to the data query request, the node health status of each of the databases is obtained.
[0023] Further, the obtaining of the node health status of each of the databases includes:
[0024] Obtaining the working status of each of the database nodes, the nodes including intermediate management nodes, historical nodes, routing nodes, query nodes, overlord nodes and coordination nodes;
[0025] The health status of the node is obtained according to the working status of the node.
[0026] Furthermore, obtaining the health status of the node according to the working status of the node includes:
[0027] When the working status of the intermediate management node, the historical node, the routing node, the query node, the overlord node and the coordination node are all online, the node health status is a normal working status;
[0028] When one or more of the intermediate management node, the historical node, the routing node, and the query node are offline, the node health state is a sub-healthy state;
[0029] When the overlord node and / or the coordination node is offline, the node health status is a downtime status.
[0030] Specifically, the obtaining of the health status of the multiple databases further includes:
[0031] The business processing volume health status and hardware health status of each of the databases are obtained.
[0032] Specifically, generating a query priority order according to the health status of each of the databases includes:
[0033] The query priority is generated according to the node health status, the data health status, the business processing volume health status and the hardware health status.
[0034] On the other hand, this article also provides a Druid database data query device, including:
[0035] A receiving unit, used for receiving a data query request from a user;
[0036] An acquisition unit, configured to acquire health status of multiple databases according to the data query request, wherein the databases are respectively deployed in different parks;
[0037] A query priority generation unit, used to generate a query priority according to the health status of each of the databases;
[0038] A query unit, configured to select the database with the best health status to perform data query according to the query priority order, and obtain a query result;
[0039] The feedback unit is used to feed back the query results to the user.
[0040] On the other hand, this document also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method steps described in the above technical solution when executing the computer program.
[0041] By adopting the above technical scheme, the Druid database data query method, device and computer equipment described in this article can meet the data query needs of Druid databases deployed across parks, and can obtain the health status of the Druid databases in each park according to the user's data query request, so as to send the data query business to the database with the best health status for processing, thereby greatly avoiding query failures caused by query result jitter and improving query quality.
[0042] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0044] Figure 1 A schematic diagram of a Druid database data query method provided in an embodiment of this invention is shown;
[0045] Figure 2 A flow chart of a method for obtaining the data health status of a database provided in an embodiment of this invention is shown;
[0046] Figure 3 Another method flow chart for obtaining data health status provided by an embodiment of this invention is shown;
[0047] Figure 4 A flow chart of a method for obtaining the health status of a database node provided in an embodiment of this invention is shown;
[0048] Figure 5 A structural schematic diagram of a Druid database data query device provided in an embodiment of this invention is shown;
[0049] FIG6( a ) shows a schematic diagram of an application scenario of a Druid database data query method;
[0050] FIG6( b ) shows another application scenario schematic diagram of the Druid database data query method;
[0051] Figure 7 A schematic diagram of a computer device provided in an embodiment of this invention is shown.
[0052] Description of the accompanying symbols:
[0053] 51. Receiving unit;
[0054] 52. Get unit;
[0055] 53. Query priority generation unit;
[0056] 54. Query unit;
[0057] 55. Feedback unit;
[0058] 702. Computer equipment;
[0059] 704, processor;
[0060] 706. Memory;
[0061] 708, driving mechanism;
[0062] 710, input / output module;
[0063] 712. Input devices;
[0064] 714. Output devices;
[0065] 716. Presentation equipment;
[0066] 718. Graphical user interface;
[0067] 720, network interface;
[0068] 722. Communication link;
[0069] 724. Communication bus. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of this article to clearly and completely describe the technical solutions in the embodiments of this article. Obviously, the described embodiments are only part of the embodiments of this article, not all of the embodiments. Based on the embodiments of this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article.
[0071] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0072] Druid is a distributed database that is widely used because it can provide data query services with ultra-large data scale and millisecond latency, as well as excellent performance in data query with high real-time requirements. Druid is usually deployed in a single park (also called a data center). However, cross-park deployment can greatly improve Druid's high availability. Therefore, cross-park Druid deployment has gradually attracted people's attention. In the prior art, cross-park Druid deployment usually adopts the following methods: 1. All Druid nodes are distributed and deployed in multiple parks; 2. Druid clusters are deployed in two parks respectively, and then the two Druid clusters are proxied by proxy devices. However, both deployment methods have some defects. Specifically, in method 1, during the data ingestion stage, Druid works in multiple nodes, and a merge operation is required before writing to deep storage, which seriously wastes cross-campus network bandwidth, increases time consumption, and further increases the pressure on management nodes, which can easily cause a vicious cycle and cause the entire cluster to crash. In addition, Druid's dependent modules, such as HDFS (Hadoop Distributed FileSystem), do not support cross-campus deployment. If these modules crash, the entire Druid cluster will crash, making Druid's high availability insufficient. In method 2, since the proxy device can only detect whether the Druid cluster is in a downtime state, when some of the nodes are offline, it will still assume that Druid can work normally, and then poll the Druid cluster to complete the data query business. At this time, although the Druid cluster has not crashed, data may appear from time to time, resulting in poor query results. Polling means that when the agent learns that both Druid clusters are not down, it sends the current data query request business to a Druid cluster that did not execute the data query business last time, that is, the two Druid clusters take turns to execute data queries.
[0073] In order to solve the above problems, the embodiments of this article provide a Druid database data query method, which can improve the accuracy of data query and improve user satisfaction. Figure 1 It is a flowchart of a Druid database data query method provided in an embodiment of this article. The method is applicable to a Druid database system deployed across campuses, and the system includes at least two campuses, each of which is equipped with a Druid database. It should be noted that this specification provides method operation steps as described in the embodiments or flow charts, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or device product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or drawings. Specifically, Figure 1 As shown, the method may include:
[0074] S101: receiving a data query request from a user;
[0075] S102: Acquire health status of multiple databases according to the data query request, where the multiple databases are deployed in different parks respectively;
[0076] That is, in the embodiment of the present specification, the Druid database is deployed across parks, and each park is deployed with a Druid database. In addition, as shown in Figures 6(a) and 6(b), which are schematic diagrams of application scenarios of the query method provided in the embodiment of the present specification, each park is also equipped with a Druid database dependency module, including zookeeper, which acts as a registration center, stores metadata of the Druid database, and maintains the operation of the druid cluster; HDFS, which acts as deep storage to achieve cold data disaster recovery backup, and can also be used to transfer data; MySQL, which is a relational database management system that provides Druid database management console queries; and some other dependencies not shown in the accompanying drawings, such as Kafka, which provides Druid database data ingestion.
[0077] S103: generating a query priority order according to the health status of each of the databases;
[0078] S104: According to the query priority order, select the database with the best health status to perform data query to obtain query results;
[0079] That is, the data of each park is sorted from best to worst according to their respective health status, and the data query service is sent to the database corresponding to the park with the best health status;
[0080] S105: Feedback the query result to the user.
[0081] A Druid database data query method provided in an embodiment of this specification is suitable for a Druid database deployed across parks. After receiving a data query request from a user, the health status of the Druid database in each park is obtained according to the specific data query request, so as to send the data query service to the database with the best health status for processing, which can greatly avoid query result jitter and improve query quality.
[0082] It should be noted that in the embodiments of this specification, the data ingestion of the databases of the various parks is the same, that is, the data pre-stored in the Druid databases of the various parks are the same, and the servers of the databases of the various parks are equal. Since the Druid database can be deployed across parks in the embodiments of this specification, the high availability of the Druid database system can be greatly improved compared to the existing technology of distributing all Druid nodes in multiple parks.
[0083] The health status of the database includes the health status of the data in the database, such as Figure 2 As shown, a flow chart of a method for obtaining the data health status of a database is shown, and step S102: obtaining the health status of multiple databases according to the data query request, specifically includes:
[0084] S201: parsing the data query request to obtain a first data storage unit required by the data query request;
[0085] That is, the first data storage unit corresponds to the user's data query request. In the Druid database, data is stored in the data storage unit (also called a table). For example, the user's data query request is to query employee information, and the data required for the data query request is parsed and located in the data storage unit a.
[0086] In some embodiments, the data query request can be parsed to obtain the time information required for the data query request. For example, for a data query request to "query the information of employees who joined the company in 2020", the time information can be used to filter the data storage units, and there is no need to filter the data storage units with a filing time earlier than 2020, which is conducive to improving the response speed to the data query request.
[0087] S202: Counting first dimension cardinality information of the first data storage unit corresponding to each database;
[0088] Since the data ingestion tasks of each database are the same in advance, the first dimension cardinality information of the data storage unit a in each database is counted at this time.
[0089] For example, data storage unit a is an employee information statistics table, which stores information such as time column, dimension column and indicator column. The time column represents the timestamp of table a, which can be used to represent the filing time of data storage unit a; the dimension column includes employee information, such as employee number information, employee name information, employee age information, department information, etc.; the indicator column represents the numbers that need to be aggregated, such as the number of successes, the number of failures, the number of clicks, etc., which can be used to represent the number of times the data storage unit a is referenced or queried.
[0090] The first dimension cardinality information of the data storage unit is counted, specifically the number of dimension columns or the number of permutations and combinations thereof in the data storage unit a is counted, and the first dimension cardinality information can be used to characterize the richness of the data stored therein. In addition, it can also be judged based on the volume of the data storage unit.
[0091] S203: Obtaining the data health status of the database according to the first dimension cardinality information of each database.
[0092] For example, the query statistics of the database of one of the parks (referred to as the first park, database A) show that the first dimension cardinality of its data storage unit a is 4, and the corresponding data storage unit a in the database of another park (referred to as the second park, database B) has lost the employee name information, and the first dimension cardinality is 3. The health status of each database can be obtained according to the first dimension cardinality information of the data storage unit a in each database, for example, the data health status of the database can be scored and rated according to the first dimension cardinality information.
[0093] Then, when generating the query priority of each database according to the health status of the database, specifically the data health status, the first dimension cardinality values of each database can be compared: since the first dimension cardinality corresponding to database A is 4, which is greater than the first dimension cardinality corresponding to database B, which is 3, it can be known that the data health status of database A is better than the data health status of database B, and the query priority is: database A takes precedence over database B. It should be noted that the number of the parks and their corresponding databases can be more than two, and only database A and database B are used as examples here, and the case of more databases is similar to the case of two databases.
[0094] It should be noted that, in the embodiments of this specification, statistics on the data storage unit to obtain the first dimension cardinality information can be implemented through the query node (Broker) of the Druid database.
[0095] Therefore, in the embodiment of the present specification, the data health status of each database may be sorted according to the value of the first dimension cardinality of each database, so that the data query request is sent to a database with the best data health status for execution.
[0096] In some possible embodiments, the first data storage unit may include multiple data storage units, that is, multiple data storage units need to be combined to meet the user's data query requirements. For example, the required data includes data stored in data storage unit a and data stored in data storage unit b. Preferably, the first dimension cardinality information of the two data storage units is queried for database A and database B respectively. If it is found that the information of data storage unit a in database A is more complete than that of database B, and the information of data storage unit b is missing, the data of data storage unit a is queried from database A and the data of data storage unit b is queried from database B to jointly complete the user's data query request.
[0097] A Druid database data query method provided in an embodiment of the present specification can analyze the health status of the first data storage unit storing the required data in each database according to the user data query request, so as to use a database or a combination of databases with more comprehensive data information to complete the data query business, so that the data query results can be more accurate and the data query quality can be improved.
[0098] like Figure 3 As shown, in some preferred embodiments, obtaining the health status of multiple databases may also include:
[0099] S301: Acquire the second data storage unit of each database according to a preset screening rule;
[0100] That is, the second data storage unit corresponds to a preset screening rule, and the preset screening rule may be to screen by the archiving time of the data storage unit, for example, to randomly generate a time period condition at every time interval, and obtain one or more data storage units corresponding to each database that meet the time period condition. Of course, there may be multiple preset screening rules, for example, to screen by the items corresponding to the data storage unit, to screen by the source of the data stored in the data storage unit, and so on.
[0101] S302: Counting the second dimension cardinality information of the second data storage unit corresponding to each database;
[0102] That is, the second data storage unit obtained in step S301 is counted in units of databases. For example, for the second data storage unit that meets the preset filtering condition X (the second data storage unit can be a data storage unit or composed of multiple data storage units), the number of items in the dimension column of the second data storage unit or the number of its permutations and combinations is counted, which is the second dimension cardinality information.
[0103] S303: Obtaining the data health status of each database according to the second dimension cardinality information of each database.
[0104] That is, the second dimension cardinality information corresponding to the first park database A and the second park database B is obtained respectively, and then when generating the query priority according to the health status of each database, the query priority is generated according to the specific size of the second dimension cardinality information: if the number of dimension column items in the second dimension cardinality information of the first park database A is greater than the number of dimension column items in the second dimension cardinality information corresponding to the second park database B, it means that the data health status of database A is better than that of database B.
[0105] That is, in the embodiment of the present specification, one or a group of data storage units are randomly selected to perform dimension column item statistics, so as to have a global control over the data health status of the database in real time. Therefore, when receiving a data query request sent by a user, the data query request is preferentially sent to a database with the best data health status, which can avoid sending the data query request to a database with serious data loss, and can improve the response and execution efficiency of the data query request.
[0106] The above statistics on the number of dimension column items of one or a group of data storage units to obtain the second dimension cardinality information to evaluate the data health status of the database can be executed before step S201: parsing the data query request to obtain the first data storage unit required for the data query request, and can also be executed simultaneously with step S201, and can also be executed after step S201.
[0107] In some feasible embodiments, the health status of a database includes the data health status and node health status of the database. Then step S102: obtaining the health status of multiple databases according to the data query request, in addition to querying the data health status of the database of each park, also includes:
[0108] According to the data query request, the node health status of the database corresponding to each park is obtained.
[0109] like Figure 4 As shown, obtaining the node health status of the database corresponding to each park further includes:
[0110] S401: Obtaining the working status of nodes of each database, the nodes including intermediate management nodes, historical nodes, routing nodes, query nodes, overlord nodes and coordination nodes;
[0111] Among them, the middle management node (Middle Manager) in the Druid database can be used to be responsible for data ingestion, provide real-time data query, and regularly transfer data to the historical node through HDFS (distributed file system); the historical node (Historical) can be used to read the persistent data of the middle management node from HDFS, proxy the query business of these data, and receive data loading and deletion instructions from the coordination node; the routing node (Router) proxy query node, overlord node and coordination node API, provide management page; the query node (Broker) can be used to receive query requests, analyze the location of the data required for the query request, and specify the query plan; the overload node (Overload) is responsible for the allocation and management of data ingestion tasks to achieve load balancing of data ingestion; and the coordination node (Coordinator) is used to manage data storage and achieve load balancing of historical node storage.
[0112] In the embodiments of the present specification, the working status of each node can be obtained by querying the registration information of each node in Zookeeper (node survival list), and Zookeeper can update the node survival list to monitor the working status of each node in real time.
[0113] S402: Acquire the node health status according to the node working status.
[0114] Step S402: Obtain the node health status according to the working status of the node, specifically:
[0115] When the working status of the intermediate management node, the historical node, the routing node, the query node, the overlord node and the coordination node are all online, the node health status is a normal working status;
[0116] When one or more of the intermediate management node, the historical node, the routing node, and the query node are offline, the node health state is a sub-healthy state;
[0117] When the overlord node and / or the coordination node is offline, the node health status is a downtime status.
[0118] Sort by the node health status of the database corresponding to each park. Normal working status is better than sub-health status, and sub-health status is better than downtime status.
[0119] When there are a large number of parks, there may be multiple parks whose database nodes have the same health status, for example, all are in normal working status, or all are in sub-healthy status; and the Druid database data query method provided in the embodiment of this specification can sort the databases based on the node health status and the data health status, so that the database with the best health status in all aspects can be selected to execute the data query business, thereby improving the quality of data query service.
[0120] If there are still two or more databases whose node health status and data health status are the same, then further, the embodiment of this specification provides a Druid database data query method, wherein step S102: according to the data query request, obtaining the health status of the database of each of the parks, further includes:
[0121] S501: Acquire the business processing volume health status and hardware health status of each of the park databases according to the query demand information.
[0122] Among them, the business processing volume health status refers to the number of data query requests currently processed by the database corresponding to each park; for example, the business number of data query requests currently processed by the first park database A is 100, and the business number of data query requests currently processed by the second park database B is 90, which means that database A is busier than database B. Under the premise that the node health status and data health status of database A and database B are equal, the data query request newly sent by the user will be sent to the second park database B for execution.
[0123] The hardware health status refers to the server occupancy (or idleness) of each database for data query requests. For example, to respond to the current data query request, the server occupancy rate of the first park database A is 90%, and the server occupancy rate of the second park database B to respond to the data query request it is currently processing is 92%, which means that database A is more capable of responding to the new data query request, so the new data query request is sent to the first park database A for execution. The hardware health status can also be obtained by obtaining other indicators such as memory occupancy or combining multiple hardware indicators.
[0124] Therefore, preferably, in the embodiment of this specification, step S104: according to the query priority, select the database with the best health status to perform data query to obtain the query result, further comprising:
[0125] The query priority is generated according to the node health status, the data health status, the business processing volume health status and the hardware health status.
[0126] When the working status of the dependent devices of the Druid database is poor, the health status of the Druid database will also be affected. Therefore, in some preferred embodiments, the acquisition of the health status of multiple databases may also include: acquiring the health status of the dependency relationship between the Druid database and its dependent devices in each park. For example, acquiring the health status of the dependency relationship between HDFS and zookeeper, the dependency relationship between the Druid database and HDFS, etc.
[0127] In summary, the Druid database data query method provided in the embodiments of this specification can evaluate the health status of the database from multiple dimensions such as node health status, data health status, business processing volume health status and hardware health status. The evaluation of the database health status is finer in granularity and more accurate.
[0128] Since the prior art cannot obtain the detailed health status of each database through port detection (port detection can only know whether the database is down), for example, when the query node works normally but data is lost, resulting in poor data health, the user's data query request will still be sent to the database for execution, which will cause the query result to jitter, that is, the data is sometimes available and sometimes not. In the embodiment of this specification, the database can be sorted according to its health status according to the user's data query request, so that the data query request can be sent to a database with the best health status for execution. Therefore, compared with the prior art, the query method provided in this specification can enable the database with the best health status to process the query business, which can greatly improve the accuracy of the query results, improve the success rate of data query, and improve user satisfaction.
[0129] like Figure 5 As shown, the embodiment of this specification also provides a Druid database data query device, including:
[0130] A receiving unit 51 is used to receive a data query request from a user;
[0131] An acquisition unit 52 is used to acquire health status of multiple databases according to the data query request, where the databases are respectively deployed in different parks;
[0132] A query priority generation unit 53, used to generate a query priority according to the health status of each of the databases;
[0133] A query unit 54, configured to select the database with the best health status to perform data query according to the query priority order, and obtain a query result;
[0134] The feedback unit 55 is used to feed back the query result to the user.
[0135] Corresponding to an application scenario shown in Figure 6(a), an embodiment of the present specification also provides a Druid database system, the system includes at least a first park and a second park, the first park and the second park are both provided with a database and dependent devices of the database; the system is also provided with a query front-end device, and the query front-end device establishes a communication connection with the database of each park respectively.
[0136] The query pre-processing device includes a first receiving module, a first parsing module, a first monitoring module, a first proxy module and a first feedback module;
[0137] The first receiving module is used to receive a data query request from a user;
[0138] The first parsing module parses the user's data query request to obtain data query requirement information; for example, obtains the data storage unit required by the data query request.
[0139] The first monitoring module is used to obtain the health status of each database according to the data query demand information, and generate a database query priority according to the health status of each database; the health status of the database includes the node health status, the data health status, the business processing volume health status and the hardware health status.
[0140] The first proxy module is used to distribute the data query request to the database with the best health status according to the database query priority order, and obtain the query result;
[0141] The first feedback module is used to feed back the query result to the user.
[0142] Corresponding to an application scenario shown in Figure 6(b), an embodiment of the present specification also provides a Druid database system, the system includes at least a first park and a second park, the first park and the second park are both equipped with a database, a query front-end device and dependent devices of the database; the query front-end device establishes a communication connection with the database of each corresponding park, and the query front-end devices belonging to different parks are communicated with each other.
[0143] The query pre-processing device includes a second receiving module, a second parsing module, a second monitoring module, a second proxy module, a second feedback module and a communication module;
[0144] The second receiving module is used to receive a data query request from a user;
[0145] The second parsing module parses the user's data query request to obtain data query requirement information;
[0146] The second monitoring module is used to obtain the health status of each database according to the data query demand information;
[0147] The communication module is used to interact with the query front-end devices of other parks to obtain the health status of other databases;
[0148] The second monitoring module is also used to generate a database query priority based on the health status of each database obtained through interaction with the communication module; the health status of the database includes the node health status, the data health status, the business processing volume health status and the hardware health status.
[0149] The second proxy module is used to distribute the data query request to the database with the best health status according to the database query priority order, and obtain the query result;
[0150] The second feedback module is used to feed back the query result to the user.
[0151] The Druid database system provided in the embodiments of this specification is provided with a query front-end device, which not only realizes the high availability of the database, but also can generate preferential data according to the health status of each database in response to the user's data query request, so that the database with the best health status can execute the query business, avoid the jitter of the query result and improve the accuracy of the query result.
[0152] The Druid database system provided in the embodiments of this specification is configured with dependent devices in each park, so it can avoid the security and reliability issues caused by cross-park dependent devices when building a Druid database in multiple parks, and can ensure the data ingestion speed, which is conducive to reducing the workload of intermediate management nodes.
[0153] like Figure 7As shown, a computer device provided in an embodiment of the present invention, the computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 702 may also include any memory 706, which is used to store any kind of information such as code, settings, data, etc. Non-restrictive, for example, the memory 706 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 702. In one case, when the processor 704 executes an associated instruction stored in any memory or a combination of memories, the computer device 702 may perform any operation of the associated instruction. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0154] The computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input device 712) and for providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface 718 (GUI). In other embodiments, the input / output module 710 (I / O), input device 712, and output device 714 may not be included, and the computer device 702 may be used as a computer device in a network. The computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.
[0155] The communication link 722 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.
[0156] Corresponds to Figure 1-Figure 4 The method in this article also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.
[0157] The embodiment of the present invention also provides a computer readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 4The method shown.
[0158] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.
[0159] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0160] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.
[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0162] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.
[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.
[0164] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0166] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.
Claims
1. A Druid database data query method, characterized in that: The method comprises: Receive data query requests from users; According to the data query request, the health status of multiple databases is obtained, and the multiple databases are respectively deployed in different parks; including: parsing the data query request to obtain the first data storage unit required by the data query request; counting the first dimension cardinality information of the first data storage unit corresponding to each database; and obtaining the data health status of each database according to the first dimension cardinality information of each database; generating a query priority order according to the health status of each of the databases; According to the query priority order, select the database with the best health status to perform data query to obtain the query result; Feedback the query results to the user.
2. A Druid database data query method according to claim 1, characterized in that: The obtaining of the health status of the multiple databases also includes: Acquire the second data storage unit of each of the databases according to preset screening rules, wherein the preset screening rules include screening by the archiving time of the data storage unit, screening by the project corresponding to the data storage unit, and screening by the source of the data stored in the data storage unit; Counting the second dimension cardinality information of the second data storage unit corresponding to each database; The data health status of each database is obtained according to the second dimension cardinality information of each database.
3. A Druid database data query method according to claim 1, characterized in that: The obtaining the health status of the plurality of databases according to the data query request further includes: According to the data query request, the node health status of each of the databases is obtained.
4. A Druid database data query method according to claim 3, characterized in that: The obtaining of the node health status of each database includes: Obtaining the working status of each of the database nodes, the nodes including intermediate management nodes, historical nodes, routing nodes, query nodes, overlord nodes and coordination nodes; The health status of the node is obtained according to the working status of the node.
5. A Druid database data query method according to claim 4, characterized in that: The obtaining the health status of the node according to the working status of the node includes: When the working status of the intermediate management node, the historical node, the routing node, the query node, the overlord node and the coordination node are all online, the node health status is a normal working status; When one or more of the intermediate management node, the historical node, the routing node, and the query node are offline, the node health state is a sub-healthy state; When the overlord node and / or the coordination node is offline, the node health status is a downtime status.
6. A Druid database data query method according to claim 3, characterized in that: The obtaining of the health status of the multiple databases also includes: The business processing volume health status and hardware health status of each of the databases are obtained.
7. A Druid database data query method according to claim 6, characterized in that: Generating a query priority order according to the health status of each of the databases further includes: The query priority is generated according to the node health status, the data health status, the business processing volume health status and the hardware health status.
8. A Druid database data query device, characterized in that: include: A receiving unit, used for receiving a data query request from a user; An acquisition unit is used to acquire the health status of multiple databases according to the data query request, wherein the databases are respectively deployed in different parks, including parsing the data query request to obtain a first data storage unit required by the data query request; Counting the first dimension cardinality information of the first data storage unit corresponding to each database; and obtaining the data health status of each database according to the first dimension cardinality information of each database; A query priority generation unit, used to generate a query priority according to the health status of each of the databases; A query unit, configured to select the database with the best health status to perform data query according to the query priority order, and obtain a query result; The feedback unit is used to feed back the query results to the user.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method steps according to any one of claims 1 to 7 are implemented.
