Multi-data-source data query method and device, computer equipment and storage medium

Abstract

The invention relates to a multi-data-source data query method and device, computer equipment and a storage medium. The method comprises the steps of obtaining a data query task, wherein the data query task comprises a view identifier; screening a target view configuration table from a plurality of pre-configured view configuration tables according to the view identifier, the target view configuration table comprising query configuration information, and the query configuration information comprising a plurality of data source identifiers; acquiring multi-data source configuration information; wherein the multi-data-source configuration information comprises connection information among a plurality of data sources and query rules and execution logics of the data sources; splitting the data query task according to the plurality of data source identifiers and the data source configuration information to generate a plurality of sub-query tasks respectively adaptive to different data sources; and distributing each sub-query task to each data source for data query. By adopting the method, the flexibility and expansibility of a data query system can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer software technology, and in particular to a method, device, computer equipment and storage medium for querying data from multiple data sources. Background Art

[0002] With the development of computer software technology, data query technology has emerged. Traditional data query methods are mostly customized for applications of specific data sources, lack versatility, and cannot easily adapt to changes in different data sources and query requirements.

[0003] In traditional technology, whenever a new data source needs to be queried, the existing code often needs to be redeveloped or significantly modified. This is because in traditional data query methods, the query logic is often embedded in the code and lacks standardized and configured management. The query logic implemented by different developers may vary greatly, making it difficult to ensure the consistency and repeatability of query operations, resulting in low development efficiency and high maintenance costs. Summary of the invention

[0004] Based on this, it is necessary to provide a multi-data source data query method, device, computer equipment and storage medium that can improve the flexibility and scalability of data query in response to the above technical problems.

[0005] In a first aspect, a method for querying data from multiple data sources is provided, the method comprising:

[0006] Get a data query task, which includes a view identifier;

[0007] Filtering a target view configuration table from a plurality of pre-configured view configuration tables according to a view identifier, wherein the target view configuration table includes query configuration information, and the query configuration information includes a plurality of data source identifiers;

[0008] Acquire multi-data source configuration information; wherein the multi-data source configuration information includes connection information between multiple data sources and query rules and execution logic of each data source;

[0009] The data query task is split according to multiple data source identifiers and data source configuration information to generate multiple sub-query tasks that are adapted to different data sources respectively;

[0010] Assign each subquery task to each data source for data query.

[0011] In some embodiments, the query configuration information also includes at least one of a configuration table name, an input parameter configuration table, an input parameter rule verification table, an output parameter configuration table, cache expiration information, and an SQL template.

[0012] In some embodiments, the query configuration information also includes engine configuration information; each sub-query task is assigned to each data source to perform data query, including:

[0013] Determine the execution engine corresponding to each subquery task according to the engine configuration information;

[0014] According to the grammatical rules of each execution engine, each sub-query task is split into multiple query statement fragments that are adapted to different execution engines respectively;

[0015] Each query statement fragment is respectively assigned to the corresponding execution engine in the corresponding data source for data query.

[0016] In some embodiments, each sub-query task is split into multiple query statement fragments adapted to different execution engines according to the grammatical rules of each execution engine, including:

[0017] Generate an AST tree based on the query language of each sub-query task;

[0018] Use RBO optimization algorithm to optimize the rules of AST tree;

[0019] According to the grammatical rules of each execution engine and the optimized AST tree, the query language of each subquery task is split into multiple query statement fragments based on the SQL query language that are adapted to different execution engines.

[0020] In some embodiments, the query configuration information further includes multiple query identifiers, and the method further includes:

[0021] In response to receiving the query results corresponding to each sub-query task, the sub-query task that needs to be queried multiple times is determined according to the multiple query identifiers, and the multiple query process is enabled for the sub-query task that needs to be queried multiple times.

[0022] In some embodiments, the query configuration information further includes cache expiration information, and the method further includes:

[0023] Receive the query results corresponding to each subquery task, bind each query result to the cache key of the corresponding data source, and store them in the cache;

[0024] In response to the underlying data source update instruction, the expired query results are deleted according to the cache expiration information and the cache key.

[0025] In some embodiments, the method further comprises:

[0026] Receive the query results corresponding to each sub-query task, integrate the query results according to the response rules of the request end of the data query task through the BFF layer, and return the integrated complete query results to the request end.

[0027] In a second aspect, a multi-data source data query device is also provided, the device comprising:

[0028] A task receiving module is used to obtain a data query task, where the data query task includes a view identifier;

[0029] A view configuration determination module, used to filter a target view configuration table from a plurality of pre-configured view configuration tables according to a view identifier, wherein the target view configuration table includes query configuration information, and the query configuration information includes a plurality of data source identifiers;

[0030] A data source configuration acquisition module is used to acquire multiple data source configuration information; wherein the multiple data source configuration information includes connection information between multiple data sources and query rules and execution logic of each data source;

[0031] The task splitting module is used to split the data query task according to multiple data source identifiers and data source configuration information to generate multiple sub-query tasks that are adapted to different data sources respectively;

[0032] The data query module is used to assign each sub-query task to each data source for data query.

[0033] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-data source data query method of the first aspect are implemented.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-data source data query method of the first aspect.

[0035] The above-mentioned multi-data source data query method, device, computer equipment and storage medium determine the corresponding target view configuration table through the view identifier carried in the data query task to obtain the data source identifier, obtain the pre-configured multi-data source configuration information, split the data query task according to the data source dimension according to the data source identifier and the multi-data source configuration information, and generate sub-query tasks adapted to different data sources, and assign the sub-query tasks to their respective corresponding data sources for data query. In this solution, since the data source identifier and data source configuration information are configurable, and the task can be split and the conversion can be performed according to the data source dimension, it is possible to flexibly expand different data sources, thereby improving the flexibility and scalability of the data query system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A diagram of an application environment of a multi-data source data query method in some embodiments;

[0037] Figure 2A schematic diagram of a flow chart of a method for querying data from multiple data sources in some embodiments;

[0038] Figure 3 It is a schematic diagram of the architecture of the data query method of multiple data sources in some application examples;

[0039] Figure 4 It is a structural block diagram of a multi-data source data query device in some embodiments;

[0040] Figure 5 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The multi-data source data query method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates through the network, the server 104 and the data source server 106.

[0043] Specifically, the server 104 obtains a data query task; wherein the data query task includes a view identifier; based on the view identifier, a target view configuration table is filtered from a plurality of pre-configured view configuration tables, wherein the target view configuration table includes query configuration information, and the query configuration information includes a plurality of data source identifiers; multiple data source configuration information is obtained; wherein the multiple data source configuration information includes connection information between the plurality of data sources and query rules and execution logic of each data source; the data query task is split according to the plurality of data source identifiers and the data source configuration information, and a plurality of sub-query tasks respectively adapted to different data sources are generated; and each sub-query task is respectively assigned to each data source (data source server 106) for data query.

[0044] The terminal 102 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 and the data source server 106 may be implemented as independent servers or a server cluster consisting of multiple servers.

[0045] In some embodiments, Figure 2 As shown, a multi-data source data query method is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0046] Step 202: Acquire a data query task, where the data query task includes a view identifier.

[0047] Among them, the query logic of each view can be pre-defined based on the view configuration table. The view configuration table is used to store basic information of each view and information such as SQL templates required for data query. For example, it can include view ID (Identification), view name, configuration table name, data source identifier, version number and other field information.

[0048] Specifically, the user may send a data query task to the server through a requesting end (terminal), wherein the data query task may include a view identifier (view ID).

[0049] Step 204: Filter a target view configuration table from a plurality of pre-configured view configuration tables according to the view identifier, wherein the target view configuration table includes query configuration information, and the query configuration information includes a plurality of data source identifiers.

[0050] Specifically, the server can filter the target view configuration table required for this data query task from multiple pre-customized view configuration tables according to the view ID, and extract query configuration information from the target view configuration table. The query configuration information may include data source identifiers of multiple configured data sources.

[0051] Exemplarily, the data source may include but is not limited to big data engine data sources such as ClickHouse and Holgorsen.

[0052] Exemplarily, the query configuration information also includes at least one of the configuration table name, input parameter configuration table, input parameter rule verification table, output parameter configuration table, cache expiration information, and SQL template. The input parameter configuration table is used to define query parameters, which may include but are not limited to parameter name, type, whether it is required, etc. The output parameter configuration table is used to define the field information of the query result, which may include but are not limited to column name, alias, whether desensitization is required, etc.

[0053] Step 206, obtaining multi-data source configuration information; wherein the multi-data source configuration information includes connection information between multiple data sources and query rules and execution logic of each data source.

[0054] Specifically, data source configuration items can be introduced into the configuration file, and connection information, query rules and execution logic can be defined for different data sources through data source configuration items, thereby obtaining multi-data source configuration information. Among them, query rules and execution logic of big data source engines such as ClickHouse and Holgorsen can be supported but not limited to.

[0055] Step 208: split the data query task according to multiple data source identifiers and data source configuration information to generate multiple sub-query tasks respectively adapted to different data sources.

[0056] Specifically, the server determines the corresponding multiple data sources according to the multiple data source identifiers, and then determines the execution order and association logic between the sub-query tasks between the data sources according to the connection information in the data source configuration information, and then splits the data query tasks according to the dimensions of the data source, and can generate sub-query tasks adapted to different data sources through the translation engine according to the query rules and execution logic corresponding to each data source.

[0057] Step 210: Allocate each sub-query task to each data source for data query.

[0058] Specifically, the server may initiate query requests to multiple data sources in an asynchronous or parallel manner, and assign each sub-query task to a corresponding data source to perform data query. Furthermore, data integration may be performed after the query is completed.

[0059] In the above-mentioned multi-data source data query method, the corresponding target view configuration table is determined by the view identifier carried in the data query task to obtain the data source identifier, and the pre-configured multi-data source configuration information is obtained. According to the data source identifier and the multi-data source configuration information, the data query task is split according to the data source dimension and sub-query tasks adapted to different data sources are generated, and the sub-query tasks are assigned to their respective corresponding data sources for data query. In this solution, since the data source identifier and the data source configuration information are configurable, and the task can be split and the conversion can be performed according to the data source dimension, it is possible to flexibly expand different data sources, thereby improving the flexibility and scalability of the data query system.

[0060] In some embodiments, the query configuration information also includes engine configuration information; each sub-query task is assigned to each data source for data query, including: determining the execution engine corresponding to each sub-query task according to the engine configuration information; splitting each sub-query task into multiple query statement fragments that are respectively adapted to different execution engines according to the grammatical rules of each execution engine; and each query statement fragment is assigned to the corresponding execution engine in the corresponding data source for data query.

[0061] In this embodiment, after the data query initiated by the request end is considered to be split once according to the data source dimension, it can also be split twice according to the engine dimension. For example:

[0062] One query: According to the user-configured view configuration table and the pre-configured multi-data source configuration information, semantic splitting based on indicators / dimensions is performed. That is, a job (corresponding to a data query task initiated by the user) will be split into multiple tasks (sub-query tasks), and each task represents a logical indicator / dimensional query.

[0063] Secondary splitting: also known as engine splitting, can be performed according to the engine dimension under the indicator dimension. A Task will be split into multiple Queries (query statement fragments), and each Query represents a query for the engine.

[0064] Since data engines in the big data field are rich and diverse, they can be classified into SQL (Structured Query Language) engines, KV (Key-Value) engines, and other custom DSL (Digital Subscriber Line) engines. Therefore, if there is a cross-engine situation, it can be further split from the engine dimension under the indicator (data source indicator) dimension, so as to adapt to the grammatical rules of different engines, improve the flexibility of engine adaptation, and then improve the flexibility and scalability of data query.

[0065] In some embodiments, each sub-query task is split into multiple query statement fragments that are respectively adapted to different execution engines according to the grammatical rules of each execution engine, including: generating an AST tree (Abstract Syntax Tree) according to the query language of each sub-query task; optimizing the AST tree by using an RBO (Rule-Based Optimization) optimization algorithm; and splitting the query language of each sub-query task into multiple query statement fragments based on the SQL query language that are respectively adapted to different execution engines according to the grammatical rules of each execution engine and the optimized AST tree.

[0066] In this embodiment, the query language can be converted into an AST tree, and RBO optimization can be performed based on the AST tree, so as to improve the accuracy of generating the SQL query language and further improve the accuracy of the data query.

[0067] In some embodiments, the query configuration information also includes multiple query identifiers, and the method also includes: in response to receiving query results corresponding to each sub-query task, determining the sub-query tasks that need to be queried multiple times according to the multiple query identifiers, and enabling multiple query processes for the sub-query tasks that need to be queried multiple times.

[0068] In this embodiment, since there are many complex data query tasks in actual application scenarios, such as year-on-year and quarter-on-quarter, quantile query, etc., when encountering such data query tasks, multiple query flags can be configured, so that when the query is executed, it is determined whether the sub-query task of the corresponding data source needs to be queried multiple times by judging whether the multiple query flags are carried. Among them, the multiple times here can be two, three or more, which can be adjusted according to actual needs.

[0069] In this embodiment, since multiple calculations are performed based on the returned data after the initial query result is returned, it can ensure that the accuracy requirements of the query results in complex scenarios are met.

[0070] In some embodiments, the query configuration information also includes cache expiration information, and the method also includes: receiving the query results corresponding to each sub-query task, and binding each query result with the cache key of the corresponding data source, and storing it in the cache; responding to the underlying data source update instruction, deleting the expired query results according to the cache expiration information and the cache key.

[0071] In this embodiment, a Redis cache mechanism can be added, that is, the query results and configuration information can be stored in Redis, thereby supporting high concurrent access. Each query result is bound to the Redis Key of its corresponding data source, and an invalidation mechanism can be set. When the underlying data source is updated, the relevant cache is automatically expired to ensure the real-time nature of the query.

[0072] In some embodiments, the method also includes: receiving query results corresponding to each sub-query task, integrating each query result according to the response rules of the request end of the data query task through the BFF (Backend For Frontend) layer, and returning the integrated complete query result to the request end.

[0073] In an embodiment, a BFF layer can be added between the request end (terminal) of the data query task and the processing end (server) of the data query task. Through the BFF layer, it is possible to support the differences in data format requirements of the request ends of different business parties, integrate the query results of each sub-query task according to the response rules of each request end, and return complete query results that meet the requirements to the request end, thereby decoupling the front and back ends, improving the flexibility and scalability of the front and back end matching of data queries, and thus improving the flexibility and scalability of data queries.

[0074] Below, you can refer to Figure 3 As shown, Figure 3 The following is a schematic diagram showing the architecture of a data query method for multiple data sources in some application examples. Specifically:

[0075] The user (requester) initiates a request. After the server receives the data query task, it matches the corresponding view configuration table through the view ID carried in the data query task, and extracts the query configuration information such as the input parameter configuration table, the input parameter rule verification table, the output parameter configuration table, and the cache expiration information according to the view configuration table. The task is split according to the data source identifier and the multi-data source configuration information, and the sub-query tasks obtained after the split are placed in the task pool. Based on the configuration table information, SQL query statements that adapt to each corresponding data source are automatically generated, and the sub-query tasks are distributed to each data source through the scheduler to execute the query concurrently. Furthermore, a scheduled task can be set to access the cache expiration information at a preset time to clean up the expired cache.

[0076] It should be understood that although Figure 2 and Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 and Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0077] In some embodiments, Figure 4 As shown, a multi-data source data query device is provided, including: a task receiving module 410, a view configuration determining module 420, a data source configuration acquiring module 430, a task splitting module 440 and a data query module 450, wherein:

[0078] The task receiving module 410 is used to obtain a data query task, where the data query task includes a view identifier;

[0079] A view configuration determination module 420, configured to filter a target view configuration table from a plurality of pre-configured view configuration tables according to the view identifier, wherein the target view configuration table includes query configuration information, and the query configuration information includes a plurality of data source identifiers;

[0080] The data source configuration acquisition module 430 is used to acquire multiple data source configuration information; wherein the multiple data source configuration information includes connection information between multiple data sources and query rules and execution logic of each data source;

[0081] The task splitting module 440 is used to split the data query task according to multiple data source identifiers and data source configuration information to generate multiple sub-query tasks respectively adapted to different data sources;

[0082] The data query module 450 is used to assign each sub-query task to each data source for data query.

[0083] In some embodiments, the data query module 450 determines the execution engine corresponding to each sub-query task based on the engine configuration information; splits each sub-query task into multiple query statement fragments that are respectively adapted to different execution engines according to the grammatical rules of each execution engine; and assigns each query statement fragment to the corresponding execution engine in the corresponding data source for data query.

[0084] In some embodiments, the data query module 450 generates an AST tree based on the query language of each sub-query task; optimizes the AST tree rules using the RBO optimization algorithm; and splits the query language of each sub-query task into multiple SQL query language-based query statement fragments that are adapted to different execution engines according to the grammatical rules of each execution engine and the optimized AST tree.

[0085] In some embodiments, the data query module 450 is further used to respond to receiving query results corresponding to each sub-query task, determine the sub-query tasks that need to be queried multiple times according to multiple query identifiers, and enable multiple query processes for the sub-query tasks that need to be queried multiple times.

[0086] In some embodiments, the data query module 450 is also used to receive the query results corresponding to each sub-query task, bind each query result with the cache key of the corresponding data source, and store it in the cache; in response to the underlying data source update instruction, delete the expired query results according to the cache expiration information and the cache key.

[0087] In some embodiments, the data query module 450 is also used to receive query results corresponding to each sub-query task, integrate the query results according to the response rules of the request end of the data query task through the BFF layer, and return the integrated complete query results to the request end.

[0088] For the specific definition of the multi-data source data query device, please refer to the definition of the multi-data source data query method above, which will not be repeated here. Each module in the above-mentioned multi-data source data query device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0089] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-data source data query method is implemented.

[0090] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: obtaining a data query task, the data query task including a view identifier; filtering a target view configuration table from a plurality of pre-configured view configuration tables according to the view identifier, the target view configuration table including query configuration information, the query configuration information including a plurality of data source identifiers; obtaining multiple data source configuration information; wherein the multiple data source configuration information includes connection information between the plurality of data sources and query rules and execution logic for each data source; splitting the data query task according to the plurality of data source identifiers and the data source configuration information to generate a plurality of sub-query tasks respectively adapted to different data sources; and assigning each sub-query task to each data source for data query.

[0092] In some embodiments, when the processor executes the computer program, the following steps are also implemented: determining the execution engine corresponding to each sub-query task based on the engine configuration information; splitting each sub-query task into multiple query statement fragments that are respectively adapted to different execution engines according to the grammatical rules of each execution engine; and assigning each query statement fragment to the corresponding execution engine in the corresponding data source for data query.

[0093] In some embodiments, when the processor executes the computer program, the following steps are also implemented: generating an AST tree based on the query language of each sub-query task; optimizing the rules of the AST tree using the RBO optimization algorithm; and splitting the query language of each sub-query task into multiple query statement fragments based on the SQL query language that are adapted to different execution engines respectively according to the grammatical rules of each execution engine and the optimized AST tree.

[0094] In some embodiments, when the processor executes the computer program, the following steps are also implemented: in response to receiving the query results corresponding to each sub-query task, the sub-query tasks that need to be queried multiple times are determined according to the multiple query identifiers, and the multiple query processes are enabled for the sub-query tasks that need to be queried multiple times.

[0095] In some embodiments, when the processor executes the computer program, the following steps are also implemented: receiving the query results corresponding to each sub-query task, binding each query result with the cache key of the corresponding data source, and storing it in the cache; responding to the underlying data source update instruction, deleting the expired query results according to the cache expiration information and the cache key.

[0096] In some embodiments, when the processor executes the computer program, it also implements the following steps: receiving the query results corresponding to each sub-query task, integrating each query result according to the response rules of the request end of the data query task through the BFF layer, and returning the integrated complete query result to the request end.

[0097] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining a data query task, the data query task includes a view identifier; filtering a target view configuration table from a plurality of pre-configured view configuration tables according to the view identifier, the target view configuration table including query configuration information, the query configuration information including a plurality of data source identifiers; obtaining multiple data source configuration information; wherein the multiple data source configuration information includes connection information between the plurality of data sources and query rules and execution logic for each data source; splitting the data query task according to the plurality of data source identifiers and the data source configuration information to generate a plurality of sub-query tasks respectively adapted to different data sources; and assigning each sub-query task to each data source for data query.

[0098] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: determining the execution engine corresponding to each sub-query task based on the engine configuration information; splitting each sub-query task into multiple query statement fragments that are respectively adapted to different execution engines according to the grammatical rules of each execution engine; and assigning each query statement fragment to the corresponding execution engine in the corresponding data source for data query.

[0099] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: generating an AST tree based on the query language of each sub-query task; optimizing the rules of the AST tree using the RBO optimization algorithm; and splitting the query language of each sub-query task into multiple SQL query language-based query statement fragments that are adapted to different execution engines respectively according to the grammatical rules of each execution engine and the optimized AST tree.

[0100] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: in response to receiving the query results corresponding to each sub-query task, determining the sub-query tasks that need to be queried multiple times according to the multiple query identifiers, and enabling multiple query processes for the sub-query tasks that need to be queried multiple times.

[0101] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: receiving the query results corresponding to each sub-query task, binding each query result with the cache key of the corresponding data source, and storing it in the cache; responding to the underlying data source update instruction, deleting the expired query results according to the cache expiration information and the cache key.

[0102] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: receiving the query results corresponding to each sub-query task, integrating each query result according to the response rules of the request end of the data query task through the BFF layer, and returning the integrated complete query result to the request end.

[0103] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

[0106] It should be noted that in the embodiments of the present application, data such as user information or user data must be obtained and processed after the user's authorization and consent. When the embodiments of the present application are applied to specific products or technologies, the user's permission or consent must be obtained, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for querying data from multiple data sources, the method comprising: Acquire a data query task, wherein the data query task includes a view identifier; Filtering a target view configuration table from a plurality of pre-configured view configuration tables according to the view identifier, wherein the target view configuration table includes query configuration information, and the query configuration information includes a plurality of data source identifiers; Acquire multi-data source configuration information; wherein the multi-data source configuration information includes connection information between multiple data sources and query rules and execution logic of each of the data sources; Splitting the data query task according to the multiple data source identifiers and the data source configuration information to generate multiple sub-query tasks respectively adapted to different data sources; Each of the sub-query tasks is respectively assigned to each of the data sources to perform data query.

2. The method according to claim 1, characterized in that The query configuration information also includes at least one of a configuration table name, an input parameter configuration table, an input parameter rule verification table, an output parameter configuration table, cache expiration information, and an SQL template.

3. The method according to claim 1, characterized in that The query configuration information also includes engine configuration information; the assigning each of the sub-query tasks to each of the data sources for data query includes: Determine the execution engine corresponding to each of the sub-query tasks according to the engine configuration information; Splitting each of the sub-query tasks into a plurality of query statement fragments respectively adapted to different execution engines according to the grammatical rules of each of the execution engines; Each of the query statement fragments is respectively distributed to a corresponding execution engine in a corresponding data source for data query.

4. The method according to claim 3, characterized in that The step of splitting each of the sub-query tasks into a plurality of query statement fragments respectively adapted to different execution engines according to the grammatical rules of each of the execution engines includes: Generate an AST tree according to the query language of each sub-query task; Utilize RBO optimization algorithm to optimize the rules of the AST tree; According to the grammatical rules of each execution engine and the optimized AST tree, the query language of each sub-query task is split into a plurality of query statement fragments based on the SQL query language that are respectively adapted to different execution engines.

5. The method according to claim 1, characterized in that The query configuration information also includes a multiple query identifier, and the method further includes: In response to receiving the query results corresponding to each of the sub-query tasks, the sub-query tasks that need to be queried multiple times are determined according to the multiple query identifiers, and the multiple query processes are enabled for the sub-query tasks that need to be queried multiple times.

6. The method according to claim 1, characterized in that The query configuration information also includes cache expiration information, and the method further includes: Receive the query results corresponding to each of the sub-query tasks, bind each of the query results to the cache key of the corresponding data source, and store them in the cache; In response to the underlying data source update instruction, the expired query results are deleted according to the cache expiration information and the cache key.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Receive the query results corresponding to each of the sub-query tasks, integrate the query results according to the response rules of the request end of the data query task through the BFF layer, and return the integrated complete query results to the request end.

8. A multi-data source data query device, characterized in that: The device comprises: A task receiving module, used for acquiring a data query task, wherein the data query task includes a view identifier; A view configuration determination module, configured to filter a target view configuration table from a plurality of pre-configured view configuration tables according to the view identifier, wherein the target view configuration table includes query configuration information, and the query configuration information includes a plurality of data source identifiers; A data source configuration acquisition module, used to acquire multiple data source configuration information; wherein the multiple data source configuration information includes connection information between multiple data sources and query rules and execution logic of each of the data sources; A task splitting module, used to split the data query task according to the multiple data source identifiers and the data source configuration information, and generate multiple sub-query tasks respectively adapted to different data sources; The data query module is used to assign each of the sub-query tasks to each of the data sources for data query.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images