Data query method, device and electronic device

Abstract

The embodiments of the present application provide a data query method, device, and electronic device, relating to the field of big data technology, for selecting a data source that best matches a query request for a user in a scenario with multiple data sources. In the method, a query request is first obtained; the query request is used to query data; based on the query request and metadata, at least two data sources that match the query request are determined; a first credibility of each of the at least two data sources is determined; the first credibility of each data source is determined based on at least one of a first system load of a production environment and a first response time of each data source when processing the query request; and the query request is sent to the data source with the highest first credibility among the at least two data sources.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data query method, device, and electronic device. Background Art

[0002] In big data analysis scenarios, commonly used search engines, databases, and data warehouse tools differ in their architectural design and underlying principles, addressing different problem areas. Often, a combination of multiple databases or data warehouse tools is required to meet business needs and ensure stability. Therefore, a single piece of data often needs to be written to multiple different data sources to be used by various solutions.

[0003] In this multi-data source scenario, business teams typically need to familiarize themselves with the characteristics of each data source and the scenarios in which it is and is not suitable when selecting the appropriate one for business queries. They then categorize the business characteristics and develop solutions using the specific languages supported by the data sources. However, when business teams face solution development iterations or discover stability issues, the strong coupling between the business and the specific data source necessitates repeating this process when upgrading the data source version or replacing it with a more optimized one.

[0004] Therefore, the existing technology cannot solve the technical problem of selecting the optimal data source for the business side in the scenario of multiple data sources. Summary of the Invention

[0005] Embodiments of the present application provide a data query method, apparatus, and electronic device for selecting a data source that best matches a query request for a user in a scenario with multiple data sources.

[0006] In a first aspect, an embodiment of the present application provides a data query method, including: obtaining a query request; the query request is used to query data; determining at least two data sources that match the query request based on the query request and metadata; determining a first credibility of each of the at least two data sources; the first credibility of each data source is determined based on at least one of a first system load of a production environment and a first response time of each data source when processing the query request; and sending the query request to the data source with the highest first credibility among the at least two data sources.

[0007] Based on the above solution, when data is stored in multiple data sources, by determining the primary trustworthiness of each data source under different query scenarios and conditions, the most suitable data source can be selected for the user. Furthermore, using the production environment's system load and the data source's response time for processing query requests as the basis for evaluating the primary trustworthiness allows for a comprehensive assessment of data sources based on both performance and stability, thus avoiding potential system stability risks and, in other words, preventing system crashes.

[0008] A possible implementation method, wherein determining the first credibility of each of the at least two data sources includes: the query request includes a query scenario and a query condition; the query scenario is used to represent the business to which the data queried by the query request belongs, and the query condition includes at least one of a query time span and the number of fields required to be returned; obtaining a pre-stored first correspondence between the query scenario, query condition and stability credibility of each data source; wherein the first correspondence is determined in advance based on a second system load when each data source performs a query with the query condition under the query scenario; and based on the first correspondence, determining the stability credibility of each data source corresponding to the query scenario and the query condition as the first credibility of each data source.

[0009] Based on the above solution, the stability credibility of the data source can be used as the first credibility, that is, the data source with the highest stability can be selected for the user as the most suitable data source, which can avoid stability risks in the system.

[0010] A possible implementation method, after determining the stability credibility of each data source corresponding to the query scenario and the query condition based on the first correspondence, also includes: obtaining a pre-stored second correspondence between the query scenario, query condition and performance credibility of each data source; wherein the second correspondence is pre-determined based on a second response time for each data source to obtain a response when querying the query condition under the query scenario; determining the performance credibility of each data source corresponding to the query scenario and the query condition based on the second correspondence; and determining the first credibility of each data source based on the stability credibility of each data source and the performance credibility of each data source.

[0011] Based on the above solution, determining the first credibility based on stability credibility and performance credibility allows users to select the most suitable data source based on the actual production environment. This means that when the system load is low, a high-performance data source can be selected to provide a good user experience. When the system load is high, selecting a high-stability data source can avoid system crashes caused by excessive pursuit of performance.

[0012] A possible implementation method, after determining the performance credibility of each data source corresponding to the query scenario and the query condition based on the second correspondence, also includes: determining the first response time when the system load of the production environment is lower than a first threshold; the first response time is the response time of each data source when processing one or more query requests within a specified time; determining a correction value based on the first response time and the second response time; and correcting the performance credibility based on the correction value.

[0013] Based on the above solution, the accuracy of the performance reliability can be improved by correcting the performance reliability when the system load in the production environment is low.

[0014] A possible implementation method is to determine the first credibility of each data source based on the stability credibility of each data source and the performance credibility of each data source, specifically including: determining the weight of the performance credibility and the weight of the stability credibility based on the first system load of the production environment; and performing weighted summation processing on the performance credibility and the stability credibility to determine the first credibility of each data source.

[0015] Based on the above scheme, the weighted summation of performance credibility and stability credibility can be implemented to provide users with a good experience by selecting a data source with good performance when the system load is low, and to avoid system stability risks by selecting a data source with high stability when the system load is high. In other words, it can avoid system crashes caused by excessive pursuit of performance.

[0016] A possible implementation method is to determine the weight of the performance credibility and the weight of the stability credibility based on the first system load of the production environment, and further include: when the system load of the production environment is greater than or equal to a second threshold, determining that the weight of the stability credibility is greater than the weight of the performance credibility.

[0017] Based on the above solution, when the system load is high, by increasing the weight of stability credibility, system crash can be avoided and the system can run stably.

[0018] In a possible implementation, the query scenario includes one or more of full-text search, conditional filtering, aggregate statistics, and association analysis.

[0019] Based on the above solution, since each data source is applicable to different query scenarios, by considering multiple query scenarios separately, the user's query request can be sent to the most suitable data source.

[0020] In a second aspect, an embodiment of the present application provides a data query device, comprising: a communication unit and a processing unit;

[0021] A communication unit is configured to: obtain a query request; the query request is used to query data; a processing unit is configured to: determine at least two data sources that match the query request based on the query request and metadata; determine a first credibility of each of the at least two data sources; the first credibility of each data source is determined based on at least one of a first system load of a production environment and a first response time of each data source when processing the query request; the communication unit is further configured to: send the query request to the data source with the highest first credibility among the at least two data sources.

[0022] In one possible implementation, the processing unit determines the first credibility of each of the at least two data sources, and is specifically configured as follows: the query request includes a query scenario and a query condition; the query scenario is used to indicate the business to which the data queried by the query request belongs, and the query condition includes at least one of a query time span and the number of fields required to be returned; a first correspondence between the query scenario, query condition and stability credibility of each data source stored in advance is obtained; wherein the first correspondence is determined in advance based on the second system load when each data source performs a query with the query condition under the query scenario; based on the first correspondence, the stability credibility of each data source corresponding to the query scenario and the query condition is determined as the first credibility of each data source.

[0023] In a possible implementation, after the processing unit determines the stability credibility of each data source corresponding to the query scenario and the query condition based on the first correspondence, the processing unit is further configured to: obtain a pre-stored second correspondence between the query scenario, query condition and performance credibility of each data source; wherein the second correspondence is pre-determined based on a second response time for each data source to obtain a response when querying the query condition under the query scenario; determine the performance credibility of each data source corresponding to the query scenario and the query condition based on the second correspondence; and determine the first credibility of each data source based on the stability credibility of each data source and the performance credibility of each data source.

[0024] In one possible implementation, after the processing unit determines the performance credibility of each data source corresponding to the query scenario and the query condition based on the second correspondence, the processing unit is further configured to: determine the first response time when the system load of the production environment is lower than a first threshold; the first response time is the response time of each data source when processing one or more query requests within a specified time; determine a correction value based on the first response time and the second response time; and correct the performance credibility based on the correction value.

[0025] In one possible implementation, the processing unit determines the first credibility of each data source based on the stability credibility of each data source and the performance credibility of each data source, and is specifically configured to: determine the weight of the performance credibility and the weight of the stability credibility based on the first system load of the production environment; and perform weighted summation processing on the performance credibility and the stability credibility to determine the first credibility of each data source.

[0026] In one possible implementation, the processing unit determines the weight of the performance credibility and the weight of the stability credibility based on the first system load of the production environment, and is also configured to: when the system load of the production environment is greater than or equal to a second threshold, determine that the weight of the stability credibility is greater than the weight of the performance credibility.

[0027] In a possible implementation, the query scenario includes one or more of full-text search, conditional filtering, aggregate statistics, and association analysis.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0029] Memory, for storing computer instructions;

[0030] A processor is connected to the memory, and is used to execute computer instructions in the memory, and implement the method as described in the first aspect or the second aspect when executing the computer instructions.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising:

[0032] The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to perform the method according to any one of the first aspect or the second aspect.

[0033] For each of the above-mentioned aspects from the second to the fourth aspects and the technical effects that may be achieved by each of the aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible solutions in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application.

[0035] Figure 1 A system for applying a data query method provided in an embodiment of the present application;

[0036] Figure 2 One of the exemplary flow charts of a data query method provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of a typical query scenario provided in an embodiment of the present application;

[0038] Figure 4 A first corresponding relationship diagram provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of a first credibility calculation method provided in an embodiment of the present application;

[0040] Figure 6 A functional diagram of a query engine provided in an embodiment of the present application;

[0041] Figure 7 One of the exemplary flow charts of a data query method provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of a device provided in an embodiment of the present application;

[0043] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, the professional terms involved in the embodiments of the present application are introduced below.

[0045] (1) Metadata, also known as intermediary data or relay data, is data that describes data, primarily information describing data attributes, used to support functions such as indicating storage location, historical data, resource search, and file records. Metadata is a type of electronic catalog. To achieve the purpose of cataloging, it can describe and collect the content or characteristics of data, thereby facilitating data retrieval. Metadata is information about the organization, data domains, and their relationships. In short, metadata is data about data.

[0046] (2) Benchmark testing refers to the quantitative and comparable testing of a certain performance indicator of a class of test objects through the design of scientific test methods, test tools and test systems.

[0047] (3) Abstract syntax tree is an abstract representation of the syntax structure of source code in computer science. It represents the syntax structure of a programming language in a tree-like form, and each node in the tree represents a structure in the source code.

[0048] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0049] The terms "first" and "second" in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of its variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited thereto.

[0050] In addition, the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0051] Currently, existing query routing technology involves query engines storing metadata for multiple data sources. This metadata includes information such as the database tables and fields of each data source. Therefore, after parsing the input query request and comparing it with the metadata, the query request can be routed to the matching data source. However, this only addresses situations where data is stored in a single data source. When a user query request encounters multiple matching data sources, the only solution is to randomly select a data source or report an error, but it cannot provide the user with the most suitable data source. Therefore, how to provide users with the most suitable data source in a multi-data source scenario is an urgent problem to be solved.

[0052] In view of this, embodiments of the present application provide a data query method, device, and electronic device that can be applied to a big data analysis platform or query engine. The method includes: first, determining multiple matching data sources based on a query request input by a user; then, selecting the most matching data source from the matching data sources; and sending the user's query request to the most matching data source. This method is used to address the problem that existing technologies cannot provide users with the most matching data source in scenarios with multiple data sources.

[0053] See also Figure 1 A system for applying a data query method provided in an embodiment of the present application. The data query method provided in an embodiment of the present application can be applied to a system with a data query function, such as a big data analysis system. System 100 includes: a terminal device 101, a query engine 102, and a server 103. The terminal device 101 can communicate with the server through a network connection or the like, and is used to send the user's query request to the query engine 102. The query engine 102 is used to determine the data source that best matches the user's query request, and send the user's query request to the best-matching data source. The server 103 may include multiple different data sources for storing user data, and the user data stored in each data source may be the same. Among them, different data sources have different architectural designs and underlying principles, and the areas in which they solve problems are also different. Data sources may include Elasticsearch, Hive, Clickhouse, and PostgreSQL.

[0054] See Figure 2 This is one of the exemplary flow charts of a data query method provided in an embodiment of the present application. The data query method provided in an embodiment of the present application can be applied to a client of a query engine and may include the following process:

[0055] S201: The client obtains a query request.

[0056] For example, the client may obtain the query request by responding to an operation in which a user inputs the query request.

[0057] The above query request is used to query data, and the query request may include a query scenario and a query condition.

[0058] See also Figure 3 Schematic diagram of a typical query scenario provided by an embodiment of the present application. Query scenarios may include: full-text search, conditional filtering, aggregate statistics, and association analysis. Because different data sources are applicable to different query scenarios, the client can send the user's query request to one of Data Source 1, Data Source 2, Data Source 3, and Data Source 4 based on the query scenario in the query request.

[0059] The query conditions may include the query time span and / or the number of fields to be returned. For example, the query time span may be divided into data within seven days and data exceeding seven days, and the fields to be returned may include data with less than or equal to 50 fields and data with more than 50 fields. This application is not limited to this. It should be understood that the query conditions may also include other conditions, which are not limited to this application.

[0060] For example, S201 can be used to query data containing AAAA in the data within seven days in a full-text search scenario. Then it can be considered that the query scenario included in the query request in S201 is full-text search, and the query conditions include within 7 days and AAAA.

[0061] S202: The client determines at least two data sources matching the query request based on the query request and metadata.

[0062] The client can identify the data source that matches the query request based on the table and field names of each data source in the metadata. For example, the client can identify the data source with the same table and field names as the query request based on the table and field names in the metadata.

[0063] To facilitate user queries, the user's query request can be written in a specific computer language. For example, it can be Structured Query Language (SQL) or Language Integrated Query (LINQ), etc., which are not limited in this application. The embodiments of this application are described using the example of a user's query request written in SQL.

[0064] After receiving a user's query request, the client parses the SQL statement and generates an Abstract Syntax Tree (AST). The matching data source is determined by comparing the table and field names stored in the SQL Identifier nodes in the AST with the table and field names of each data source in the metadata. The metadata is pre-stored on the client.

[0065] For example, if a user enters the SQL statement "select id from table_1_3 where name = 'es'" into the client, the SQL statement can be parsed by the parser, generating an AST to obtain SQL node (SqlNode) fields. The table and field names stored in the nodes of type SqlIdentifier are compared with the table and field names of each data source in the metadata to determine one or more data sources that match the SQL statement.

[0066] S203: The client determines a first credibility of each of the at least two data sources.

[0067] The first credibility of each data source is determined based on at least one of a first system load of the production environment and a first response time of each data source when processing a query request. The production environment can be either the production environment of the client or the production environment of the device to which the data source belongs. The production environment refers to the online environment, i.e., the client environment actually used by users.

[0068] In a possible implementation, the first credibility of each data source may be the stability credibility of each data source, wherein the stability credibility is determined based on a pre-stored first correspondence between the query scenario, query condition, and stability credibility of each data source.

[0069] In another possible implementation, the first credibility of each data source may be determined by a weighted sum of the stability credibility and performance credibility of each data source, wherein the performance credibility is determined based on a pre-stored second correspondence between the query scenario, query conditions, and performance credibility of each data source.

[0070] S204: The client sends a query request to a data source with the highest first credibility among the at least two data sources.

[0071] Based on the first credibility obtained in S203, the client determines that the data source with the highest first credibility is the data source that best matches the query request in S201, and sends the query request in S201 to the data source that best matches the query request in S201.

[0072] In one possible implementation, the first correspondence in S203 can be obtained in the following manner: on the same terminal device, run each data source for each query scenario and query condition respectively, and then determine the second system load at this time. The stability credibility of each data source can be determined by the level of the second system load, thereby obtaining the first correspondence. For example, when the query scenario of the query request in S201 is full-text retrieval and the query condition is data within seven days, run each data source simultaneously on the same terminal device to query and determine the system load at this time. The stability credibility of a data source with a relatively low system load is high, and the stability credibility of a data source with a relatively high system load is low.

[0073] Optionally, the first corresponding relationship in S203 can also be obtained in the following manner: on the same terminal device, in each query scenario and query condition, a stability benchmark test is performed on each data source respectively, and the stability credibility of each data source is determined by the results of the benchmark test, thereby obtaining the first corresponding relationship in S203.

[0074] See also Figure 4 The first corresponding relationship diagram provided by this application. The first corresponding relationship in S203 is shown in the figure. Each data source has such a first corresponding relationship. Figure 4 Indicates the first correspondence of data source 1. If the query scenario in the query request in S201 is full-text search, and the query condition is to query data from the last 7 days, then the performance credibility of data source 1 is score 1. If the query scenario in the query request in S201 is full-text search, and the query condition is to query data from the last 7 days, and the number of fields returned is required to be less than 50, then the performance credibility of data source 1 can be half of the sum of score 1 and score 2.

[0075] Optionally, if the query scenario of the query request in S201 is full-text search and the query condition is to query data from the last 7 days, the performance credibility of data source 1 is twice that of score 1. If the query scenario of the query request in S201 is full-text search and the query condition is to query data from the last 7 days and the number of fields returned is required to be less than 50, the performance credibility of data source 1 can be the sum of scores 1 and 2.

[0076] In another possible implementation, the second correspondence in S203 can be obtained by the following method: on the same terminal device, query each query scenario and query condition through each data source respectively, and determine the second response time of the response obtained at this time. The performance credibility of each data source can be determined according to the length of the second response time, thereby obtaining the second correspondence in S203. For example, when the query scenario of the query request in S201 is full-text retrieval and the query condition is data within seven days, each data source is run simultaneously on the same terminal device to query, and the second response time of the response obtained at this time is determined. The data source with a relatively short second response time has high performance credibility, and the data source with a relatively long second response time has low performance credibility.

[0077] Optionally, the weight of the performance credibility and the weight of the stability credibility in S203 can be determined based on the first system load of the production environment. When the first system load is low, the weight of the performance credibility can be increased, and when the first system load is high, the weight of the stability credibility can be increased. In addition, when the first system load is greater than the second threshold, it is determined that the weight of the stability credibility is greater than the weight of the performance credibility. Among them, the second threshold is preset based on the situation of the production environment and experience, and this application does not limit this. That is to say, when the first system load of the production environment is low, the query request in S201 can be sent to a data source with better performance, and when the first system load of the production environment is too high, the query request in S201 can be sent to a data source with high stability.

[0078] Based on the above solution, when the system load is low, by selecting a data source with good performance, a good experience can be provided to users. When the system load is high, by selecting a data source with high stability, stability risks in the system can be avoided. In other words, system crashes caused by excessive pursuit of performance can be avoided.

[0079] In one possible implementation, when the first system load in the production environment exceeds a third threshold, the query request in S201 is placed in a waiting queue and continues execution after all query requests preceding it have been processed. When the number of query requests in the queue exceeds a fourth threshold, the query request in S201 may be rejected. The third and fourth thresholds are both preset based on the production environment and experience, and the third threshold is required to be greater than the second threshold. This application does not specifically limit the values of the third and fourth thresholds.

[0080] In one possible implementation, when the load of the first system in the production environment is low, the performance credibility of each data source can be corrected regularly. Specifically, when the load of the first system is lower than the first threshold, the client determines the first response time of each data source at this time, and then compares the first response time with the second response time to determine the correction value, and corrects the performance credibility according to the correction value. Optionally, the difference between the first response time and the second response time can be used as the correction value. The difference between the first response time and the second response time can also be multiplied by a fixed constant as the correction value. This application is not limited to this.

[0081] In one possible implementation, the matching data source determined by the client in S202 may be one or more. When there is only one matching data source, the query request in S201 can be directly sent to that data source without going through the judgment steps S203-S204. The judgment steps S203-S204 are only required when the client determines there are two or more matching data sources, and the first most credible data source, i.e., the best matching data source, is selected for the user.

[0082] See also Figure 5 A schematic diagram of a first credibility calculation method provided in an embodiment of the present application. The client can calculate the first credibility by taking a weighted sum of the stability credibility, the performance credibility, and the performance credibility correction value. Among them, the stability credibility is determined based on the second system load when the query is performed under the query scenario and query conditions in the query request. The performance credibility is determined based on the second response time of the data source when the query is performed under the query scenario and query conditions in the query request. The performance credibility correction value is determined by querying historical SQL statements when the first system load in the production environment is low, determining the first response time of each data source at this time, and then comparing the first response time with the second response time to determine the performance credibility correction value.

[0083] See Figure 6 Schematic diagram of the query engine function provided for an embodiment of the present application. As shown in the figure, first, the business party sends a query request to the query engine 601. After obtaining the query request from the business party, the query engine 601 parses and verifies the query request according to the data source metadata to determine the matching data source, which can be implemented with reference to S202. Then, based on the performance credibility, stability credibility and system load, an analysis decision is made to determine the data source that best matches the query request from the business party, which can be implemented with reference to S203. Finally, the query request from the business party is routed and forwarded to the best matching data source, that is, one of the data source A, data source B and data source C in the figure, which can be implemented with reference to S204.

[0084] Below, in order to more clearly understand the solution proposed in the embodiment of the present application, a data query method provided by the present application will be introduced in combination with specific embodiments.

[0085] See Figure 7 , which is one of the exemplary flow charts of a data query method provided in an embodiment of the present application, specifically includes:

[0086] S701, the client receives a user's query request (SQL statement).

[0087] The embodiment of the present application is described by taking the user's query request written in SQL statement as an example. S701 can be implemented with reference to S201.

[0088] S702: The client parses the user's query request to generate an AST.

[0089] The detailed process of the client parsing the user's query request to generate AST can be referred to the relevant description in S202, which will not be repeated here.

[0090] S703: The client compares the AST with the metadata to determine a matching data source.

[0091] exist Figure 7 In the embodiment shown, after the client compares the AST generated in S702 with the metadata pre-stored in the client, it is determined that the matching data sources are data source 1 and data source 2. Specifically, S703 can be implemented with reference to S202.

[0092] S704: The client calculates the first credibility of data source 1 and data source 2.

[0093] The calculation method of the first credibility of data source 1 and data source 2 can be implemented with reference to S203 and will not be repeated here.

[0094] S705 , the client sends the user's query request to data source 1 .

[0095] Assume that according to the first credibility of data source 1 and data source 2 calculated in S704 , it is determined that the first credibility of data source 1 is the highest, so the client sends the user's query request to data source 1 .

[0096] Based on the same concept of the above method, see Figure 8 , is a data query device 800 provided in an embodiment of the present application. The device 800 can perform each step in the above method. To avoid repetition, it will not be described in detail here. The device 800 includes a communication unit 801 and a processing unit 802. In one scenario:

[0097] The communication unit 801 is configured to: obtain a query request; the query request is used to query data;

[0098] The processing unit 802 is configured to perform the following processing: determining at least two data sources that match the query request based on the query request and metadata; determining a first credibility of each of the at least two data sources; the first credibility of each data source is determined based on at least one of a first system load of the production environment and a first response time of each data source when processing the query request; the communication unit 801 is further configured to send the query request to the data source with the highest first credibility among the at least two data sources.

[0099] In one possible implementation, the processing unit 802 determines the first credibility of each of the at least two data sources, and is specifically configured as follows: the query request includes a query scenario and a query condition; the query scenario is used to indicate the business to which the data queried by the query request belongs, and the query condition includes at least one of a query time span and the number of fields required to be returned; a first correspondence between the query scenario, query condition and stability credibility of each data source stored in advance is obtained; wherein the first correspondence is determined in advance based on the second system load when each data source performs a query with the query condition under the query scenario; based on the first correspondence, the stability credibility of each data source corresponding to the query scenario and the query condition is determined as the first credibility of each data source.

[0100] In one possible implementation, after the processing unit 802 determines the stability credibility of each data source corresponding to the query scenario and the query condition based on the first correspondence, it is further configured to: obtain a pre-stored second correspondence between the query scenario, query condition and performance credibility of each data source; wherein the second correspondence is pre-determined based on the second response time of each data source obtaining a response when querying the query condition under the query scenario; determine the performance credibility of each data source corresponding to the query scenario and the query condition based on the second correspondence; and determine the first credibility of each data source based on the stability credibility of each data source and the performance credibility of each data source.

[0101] In one possible implementation, after the processing unit 802 determines the performance credibility of each data source corresponding to the query scenario and the query condition based on the second correspondence, it is further configured to: determine the first response time when the system load of the production environment is lower than a first threshold; the first response time is the response time of each data source when processing one or more query requests within a specified time; determine a correction value based on the first response time and the second response time; and correct the performance credibility based on the correction value.

[0102] In one possible implementation, the processing unit 802 determines the first credibility of each data source based on the stability credibility of each data source and the performance credibility of each data source, and is specifically configured to: determine the weight of the performance credibility and the weight of the stability credibility based on the first system load of the production environment; and perform weighted summation processing on the performance credibility and the stability credibility to determine the first credibility of each data source.

[0103] In one possible implementation, the processing unit 802 determines the weight of the performance credibility and the weight of the stability credibility based on the first system load of the production environment, and is also configured to: when the system load of the production environment is greater than or equal to a second threshold, determine that the weight of the stability credibility is greater than the weight of the performance credibility.

[0104] In a possible implementation, the query scenario includes one or more of full-text search, conditional filtering, aggregate statistics, and association analysis.

[0105] Based on the same concept of the above method, see Figure 9 An electronic device is provided for an embodiment of the present application, and includes a processor 901 and a memory 902. The memory 902 is configured to store computer instructions, and the processor 901 is connected to the memory and configured to execute the computer instructions in the memory, and implement the steps of any of the above methods when executing the computer instructions.

[0106] An embodiment of the present application further provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed on a computer, the computer is caused to execute the steps of any of the above methods.

[0107] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0108] Although the specific embodiments of the present application have been described above, it should be understood by those skilled in the art that these are merely examples, and the scope of protection of the present application is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present application, but these changes and modifications all fall within the scope of protection of the present application. Although the preferred embodiments of the present application have been described, once those skilled in the art become aware of the basic creative concepts, additional changes and modifications may be made to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0109] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A data query method, characterized in that: include: Get query request; The query request is used to query data; The query request includes a query scenario and a query condition; The query scenario is used to represent the business to which the data queried by the query request belongs, and the query condition includes at least one of the query time span and the number of fields required to be returned; Determining, based on the query request and metadata, at least two data sources matching the query request; Obtaining a pre-stored first correspondence between a query scenario, a query condition, and stability credibility for each of the at least two data sources, and a second correspondence between a query scenario, a query condition, and performance credibility for each data source; wherein the first correspondence is pre-determined based on a second system load when each data source performs a query under the query condition in the query scenario; wherein the second correspondence is pre-determined based on a second response duration when each data source performs a query under the query condition in the query scenario; Determining, based on the first corresponding relationship, the stability credibility of each data source corresponding to the query scenario and the query condition; determining, according to the second corresponding relationship, the performance credibility of each data source corresponding to the query scenario and the query condition; When a first system load in the production environment is lower than a first threshold, determining a first response time; the first response time is a response time when each data source processes one or more historical query requests within a specified time; determining a correction value according to the first response time and the second response time, and correcting the performance credibility according to the correction value; determining a first credibility of each of the at least two data sources according to the stability credibility and the performance credibility of each data source; The query request is sent to the data source with the highest first credibility among the at least two data sources.

2. The method according to claim 1, characterized in that Determining the first credibility of each of the at least two data sources based on the stability credibility and the performance credibility of each data source specifically includes: Determining a weight of the performance credibility and a weight of the stability credibility according to a first system load of the production environment; A weighted summation process is performed on the performance credibility and the stability credibility to determine the first credibility of each data source.

3. The method according to claim 2, characterized in that The determining of the weight of the performance credibility and the weight of the stability credibility according to the first system load of the production environment further includes: When the first system load of the production environment is greater than or equal to a second threshold, it is determined that the weight of the stability credibility is greater than the weight of the performance credibility.

4. The method according to any one of claims 1 to 3, characterized in that The query scenarios include one or more of full-text search, conditional filtering, aggregate statistics, and association analysis.

5. A data query device, characterized in that: include: a communication unit and a processing unit; The communication unit is configured to: obtain a query request; The query request is used to query data; The query request includes a query scenario and a query condition; The query scenario is used to represent the business to which the data queried by the query request belongs, and the query condition includes at least one of the query time span and the number of fields required to be returned; A processing unit is configured to: determine at least two data sources matching the query request based on the query request and metadata; Obtaining a pre-stored first correspondence between a query scenario, a query condition, and stability credibility for each of the at least two data sources, and a pre-stored second correspondence between a query scenario, a query condition, and performance credibility for each data source; wherein the first correspondence is pre-determined based on a second system load when each data source performs a query under the query condition in the query scenario; wherein the second correspondence is pre-determined based on a second response duration when each data source performs a query under the query condition in the query scenario; Determining, based on the first corresponding relationship, the stability credibility of each data source corresponding to the query scenario and the query condition; determining, according to the second corresponding relationship, the performance credibility of each data source corresponding to the query scenario and the query condition; When a first system load in the production environment is lower than a first threshold, determining a first response time; the first response time is a response time when each data source processes one or more historical query requests within a specified time; determining a correction value according to the first response time and the second response time, and correcting the performance credibility according to the correction value; determining a first credibility of each of the at least two data sources according to the stability credibility and the performance credibility of each data source; The communication unit is further configured to send the query request to the data source with the highest first credibility among the at least two data sources.

6. An electronic device, characterized in that: include: Memory, for storing computer instructions; A processor is connected to the memory, and is configured to execute computer instructions in the memory, and implement the method according to any one of claims 1 to 4 when executing the computer instructions.

7. A computer-readable storage medium, characterized in that include: The computer-readable storage medium stores computer instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 4.

Images