A request response method, device, system and readable storage medium

By decoupling the big data fusion engine from the ROLAP cluster, the bottleneck problem of ROLAP cluster expansion capacity was solved, the system stability and processing efficiency were improved, and the needs of large-scale business volume were met.

CN114116223BActive Publication Date: 2024-11-19DUXIAOMAN TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111440837.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-30
Publication Date
2024-11-19
Estimated Expiration
2041-11-30

AI Technical Summary

Technical Problem

ROLAP clusters suffer from scalability bottlenecks when dealing with large amounts of data, resulting in insufficient system stability and reliability, and an inability to meet the demands of large-scale business operations.

Method used

The big data fusion engine is decoupled from the ROLAP cluster. Storage and computing are decoupled through the connector between the engine master node and worker nodes, allowing computing resources to scale horizontally without state and optimizing data acquisition paths through data sharding and caching.

Benefits of technology

It solves the bottleneck problem of ROLAP cluster expansion capacity, improves system stability and reliability, enhances processing efficiency, reduces the burden on the master node, and avoids the bottleneck effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114116223B_ABST
    Figure CN114116223B_ABST
Patent Text Reader

Abstract

The present invention discloses a request response method, in which a big data fusion engine receives a request and performs corresponding calculation processing, obtains data storage information through a first connector of a master control node, and then obtains data from a data source through a second connector of a working node, so that storage resources are effectively isolated from computing resources, and computing nodes can be statelessly expanded horizontally, solving the capacity bottleneck problem of online analysis scenarios; at the same time, the engine master control node only needs to undertake task parsing and request management tasks divided by execution stages, and each engine working node directly connects to a data source working node to directly obtain data, with a short data transmission path, reducing the workload of the engine master control node; at the same time, data is obtained and transmitted through fragmentation and dispersion, which can avoid problems such as the short board effect caused by obtaining a large amount of data at one time. The present invention also discloses a big data fusion engine, a system, and a readable storage medium, which have corresponding technical effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a request response method, device, system and readable storage medium. Background Art

[0002] The Online Analysis Process (OLAP) system is the most important application of the data warehouse system. It is specially designed to support complex analytical operations. It can quickly and flexibly process complex queries of large amounts of data according to the requirements of analysts, and provide query results to decision makers in an intuitive and easy-to-understand form. Traditional OLAP clusters are divided into two types: ROLAP (Relational OLAP) and MOLAP (Multi-dimension OLAP). The data organization method of ROLAP type clusters is the traditional relational database.

[0003] ROLAP clusters can solve most of the interactive analysis needs of current financial system analysts, but when the amount of data reaches a certain scale, this type of cluster has an obvious capacity bottleneck problem. This type of storage and computing coupled architecture is very costly when adding or replacing nodes, and data redistribution is required. During the redistribution process, problems such as transient network IO pressure will further appear; and the larger the amount of computing data of the node, the higher the failure rate of expansion or replacement. There are uncontrollable factors in expansion, and there will be problems such as task disconnection and retry during node synchronization. Therefore, the scale of traditional ROLAP clusters is generally within a hundred servers, which cannot meet the demand for larger-scale business volume growth.

[0004] In summary, how to solve the expansion capacity bottleneck problem of the ROLAP cluster and ensure the stability and reliability of system operation is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a request response method, device, system and readable storage medium to solve the expansion capacity bottleneck problem of the ROLAP cluster and ensure the stability and reliability of the system operation.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A request response method, comprising:

[0008] After receiving the ROLAP cluster query request initiated by the client, the big data fusion engine controls the first connector through the engine master node to obtain the surviving computing instances in the current ROLAP cluster through the ROLAP cluster master node;

[0009] After receiving the returned data source list through the first connector, the engine master control node encapsulates the data source list in data slices, and distributes the data slices to idle engine working nodes according to the cluster task allocation strategy;

[0010] The engine working node adds the loading task of the data shard to the working thread to wait for queue processing;

[0011] After receiving the data from the ROLAP cluster working node through the second connector, the engine working node returns the data to the downstream computing stage for computing and processing, so as to generate a result set;

[0012] After receiving the result set, the engine master control node feeds the result set back to the client.

[0013] Optionally, before returning the data to the downstream computing stage for computing processing, the method further includes:

[0014] The data is adapted to a data format and converted into a unified data format of the big data fusion engine.

[0015] Optionally, after the engine working node receives the data of the ROLAP cluster working node through the second connector, the method further includes:

[0016] Determine whether to configure data cache;

[0017] If data caching is configured, the data is cached in the memory of the engine worker node;

[0018] Accordingly, before the engine master node controls the first connector to obtain the surviving computing instance in the current ROLAP cluster through the ROLAP cluster master node, the method further includes:

[0019] Determine whether the acquired data source is cached data;

[0020] If so, extract the corresponding cache data from the memory of the engine working node, and execute the step of returning the data to the downstream computing stage for computing processing;

[0021] If not, the step of controlling the first connector through the engine master node to obtain the surviving computing instance in the current ROLAP cluster through the ROLAP cluster master node is executed.

[0022] Optionally, before caching the data in the memory of the engine working node, the method further includes:

[0023] Determine whether the engine working node has sufficient memory to load the data;

[0024] If yes, execute the step of caching the data into the memory of the engine working node.

[0025] Optionally, the determining whether the engine working node has sufficient memory to load the data includes:

[0026] Determine the free space and dirty data space in the engine working node;

[0027] Determine whether the total amount of the free space and the dirty data space reaches the loading amount of the data;

[0028] If it is reached, it is determined that the engine working node has enough memory to load the data; if it is not reached, it is determined that the engine working node has insufficient memory to load the data;

[0029] Accordingly, before executing the step of caching the data into the memory of the engine working node, the method further includes: clearing dirty data of the engine working node.

[0030] Optionally, after caching the data in the memory of the engine working node, the method further includes:

[0031] Determine whether the data in the ROLAP cluster is updated;

[0032] If so, a data update operation request is sent to the ROLAP cluster master node.

[0033] Optionally, before caching the data in the memory of the engine working node, the method further includes:

[0034] Determine whether the number of times the data is read within a specified time period reaches a threshold;

[0035] If so, execute the step of caching the data into the memory of the engine working node.

[0036] A big data fusion engine, comprising: an engine master control node; a first connector having one end connected to the master control node and the other end used to connect to a ROLAP master control node; an engine working node; a second connector having one end connected to the engine working node and the other end used to connect to a ROLAP working node;

[0037] The engine master control node is used to: control the computing instances that survive in the current ROLAP cluster of the first connector; after receiving the returned data source list through the first connector, encapsulate the data source list in data shards, and distribute the data shards to idle engine working nodes according to the cluster task allocation strategy; after receiving the result set, feed back the result set to the client that initiated the ROLAP cluster query request;

[0038] The engine working node is used to: add the loading task of the data shard to the working thread to wait for queue processing; after receiving the data from the ROLAP cluster working node through the second connector, return the data to the downstream computing stage for computing and processing to generate a result set.

[0039] A request response system, comprising: the big data fusion engine and the ROLAP cluster as described above;

[0040] The first connector of the big data fusion engine is connected to the master node of the ROLAP cluster, and the second connector of the big data fusion engine is connected to the working node of the ROLAP cluster.

[0041] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the request response method are implemented.

[0042] According to the method provided by the embodiment of the present invention, a big data fusion engine is connected to a data storage end, the big data fusion engine receives a request and performs corresponding calculation processing, obtains data storage information through a first connector of a master control node, and then obtains data from a data source end through a second connector of a working node, so that storage resources and computing resources are effectively isolated, and decoupling between storage and computing is achieved. Based on this, computing nodes can be statelessly expanded horizontally without being constrained or affected by a storage layer, thus solving the capacity bottleneck problem of online analysis scenarios, and making full use of computing resources and storage resources. At the same time, the connection between the computing engine and the data storage end is that the master control node connects to the master control node to obtain data storage information, and the working node connects to the working node to obtain data. The engine master control node only needs to undertake task parsing and request management tasks divided by the execution stage, and each engine working node directly connects to the data source working node to directly obtain data. Compared with the scheme in which the engine master control node connects to the data source master node to obtain data, the data transmission path is shortened, and the workload of the engine master control node is reduced. At the same time, data is obtained and transmitted in a dispersed manner through sharding, which can avoid the short board effect caused by obtaining a large amount of data at one time, and can significantly improve processing efficiency.

[0043] Correspondingly, the embodiments of the present invention also provide a big data fusion engine, system and readable storage medium corresponding to the above-mentioned request response method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 This is a flowchart of an implementation of a request response method in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of cache update in an embodiment of the present invention;

[0047] Figure 3 The figure is a connection diagram of a big data fusion engine in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The core of the present invention is to provide a request response method to solve the expansion capacity bottleneck problem of the ROLAP cluster and ensure the stability and reliability of the system operation.

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] ROLAP clusters can solve most of the interactive analysis needs of current financial system analysts, but when the amount of data reaches a certain scale, this type of cluster has an obvious capacity bottleneck problem. Therefore, the scale of traditional ROLAP clusters is generally within a hundred servers, which cannot meet the demand for larger-scale business volume increments. Therefore, how to explore new OLAP interactive analysis engines to solve the problem of dealing with the growing financial data is the main purpose of this invention.

[0051] The present invention selects Presto (a Facebook open source distributed SQL query engine, suitable for interactive analysis query, supporting GB to PB bytes of data) computing engine for secondary development and provides support from the engine side. The computing engine with decoupled computing and storage is developed to replace the current engine based on RDBMS-type storage and computing coupling, so that the computing resource nodes can support stateless expansion, and the storage can be connected to various third-party heterogeneous storage. And the computing and storage resources are fully utilized.

[0052] Presto is a distributed computing engine based on SQL developed by Facebook. It was originally used to solve the slow query problem of Hive (a data warehouse tool based on Hadoop). Presto is a pure memory-based computing engine that can meet the properties of decoupling the underlying storage and easy expansion and scalability. Users can also customize the development of connectors, data types, resource management and other components to meet the needs of senior business forms and achieve pluggable management.

[0053] The presto computing engine supports relational database connections, and business parties can query MySQL, Oracle and other RMDBS databases through presto. The fusion engine needs to solve the interoperability between the presto worker node and online query clusters such as ROLAP and MR (a programming model for parallel computing of large-scale data sets (greater than 1TB)) clusters such as Hive. The connectivity of MR clusters is already supported by the presto native computing engine, so the fusion engine needs to solve the interoperability with ROLAP clusters. This embodiment mainly introduces the interoperability method between the fusion engine and ROLAP clusters. The interconnection between the fusion engine and other databases or clusters can refer to the introduction of related technologies, which will not be repeated here.

[0054] Please refer to Figure 1 , Figure 1 This is a flowchart of a request response method in an embodiment of the present invention, the method comprising the following steps:

[0055] S101, after receiving the ROLAP cluster query request initiated by the client, the big data fusion engine controls the first connector through the engine master node to obtain the surviving computing instances in the current ROLAP cluster through the ROLAP cluster master node;

[0056] After receiving the query request from the client, the fusion engine first distributes the request to the master node, performs syntax parsing and lexical parsing of the execution plan, and then breaks it down into several calculation stages and distributes them to the working nodes; the working nodes will derive tasks to process the execution plans associated with each calculation stage.

[0057] The task determines whether the current computing stage is the final source data end (i.e., the computing stage that needs to obtain data). If so, it requests the source data. If not, it requests to be sent to the downstream computing stage.

[0058] Requesting source data specifically includes: the source data end determines the type of data source obtained, for example, if the data source is hive, requesting the hive connector to connect to the hive distributed file system; if the data source is greenplum (a data storage end based on the postgres database that provides powerful parallel data computing performance and massive data management capabilities), then connecting to the greenplum connector. In this embodiment, only the type of data source obtained is the ROLAP cluster as an example, and the request processing process is introduced from the data acquisition stage. The aforementioned process of request processing (such as the master node performing syntax parsing and division of the calculation stage, and the working node performing corresponding calculation processing) can refer to the introduction of the relevant implementation method, which will not be repeated here.

[0059] In this method, the fusion engine is directly connected to the computing node of the ROLAP cluster, and the second connector on the computing node side is transformed into a method of directly connecting to obtain data from the computing node of the ROLAP cluster. Specifically, the engine includes a master node and a computing node. The first connector is located on the master node side. After determining that the acquired data source type is a ROLAP cluster, the connector of the fusion engine first interacts with the master node (master node) of the ROLAP cluster to obtain the currently surviving computing instance information (i.e., active node) in the ROLAP cluster, such as address, port, library, schema, table and other related information. Since the data is scattered and distributed and stored in all active nodes, it is necessary to first find the active nodes, and then obtain the data separately in the active nodes. The target data can be obtained after splicing the various parts of the data.

[0060] S102: After receiving the returned data source list through the first connector, the engine master control node encapsulates the data source list in data shards, and distributes the data shards to idle engine working nodes according to the cluster task allocation strategy;

[0061] The first connector obtains information about each surviving computing node in the ROLAP cluster, effectively parses the obtained result information, assembles the acquisition connection of each computing instance data of the ROLAP cluster, and assembles the authentication of the joint engine to the ROLAP cluster into data splits;

[0062] The data shards assembled from all currently surviving computing instances in the ROLAP cluster constitute a data shard collection, which is fed back to the schedule of the engine master node. The schedule is scheduled according to the current idleness of each working node. The connector receives the returned data source list, encapsulates it in the data shard, and distributes it to the idle working nodes of the fusion engine according to the cluster task allocation strategy. Among them, the data shard scheduling of the ROLAP cluster can be dynamically adjusted according to the size of each batch of data shards scheduled by the cluster, and there is no limit on this.

[0063] S103, the engine working node adds the loading task of the data shard to the working thread and waits for queue processing;

[0064] When the working node of the fusion engine receives the allocation of data shards, it puts the loading tasks of the relevant data shards into the working thread and waits for queue processing. Among them, the processing method of the working node for the working thread can refer to the introduction of the relevant technology, which will not be repeated here.

[0065] S104, after receiving the data from the ROLAP cluster working node through the second connector, the engine working node returns the data to the downstream computing stage for computing and processing, so as to generate a result set;

[0066] Once the working node of the fusion engine receives the throughput data from the bottom-level working node of the ROLAP cluster, it returns it to the upper-level computing stage for processing. After the upper-level computing stage obtains the data, it passes it to the downstream computing stage for processing until the final result set is obtained and returned to the master node of the fusion engine.

[0067] Here, since there may be certain differences between the data directly obtained from the underlying working nodes of the ROLAP cluster and the data structure that the engine can directly recognize, and the current data migration cost between heterogeneous analysis engines is high, it is necessary to develop relevant ETL (the process of extracting, cleaning and converting the data of the business system and loading it into the data warehouse. The purpose is to integrate the scattered, messy and non-uniform data in the enterprise together to provide an analysis basis for the enterprise's decision-making) data migration between engines with completely different operations, such as batch job storage in hive and online processing jobs in mysql. When the business party conducts interactive analysis, it is necessary to migrate the data stored in other engines to the relevant ROLAP engine. This method not only makes the analysis cycle process analysis inefficient, but also causes multiple copies of the storage engine. In view of this, in order to avoid the influence of the obtained heterogeneous data sources on the data analysis process, before returning it to the upper-level computing stage for processing, the data format obtained from the ROLAP cluster can be further adapted and converted into a unified data format of the fusion engine, and then returned to the upper-level computing stage for processing.

[0068] By connecting different heterogeneous storage engines to the memory or cache layer of the big data fusion engine, eliminating the differences in storage models of the acquired heterogeneous data sources, and binding the acquired data sources to the source calculation stage of the SQL execution plan, data intercommunication between different engines is achieved, data migration is reduced, and the joint query problem between heterogeneous engine storage systems is solved.

[0069] Of course, when performing migration between non-heterogeneous data, the above settings may not be performed or other heterogeneous elimination methods may be adopted, which is not limited in this embodiment.

[0070] S105. After receiving the result set, the engine master control node feeds the result set back to the client.

[0071] The fusion engine will perform batch calculations and return the calculation results to the fusion engine's master node in batches until all result sets are returned to the client. At this point, the processing of a single request is completed.

[0072] Based on the above introduction, in the technical solution provided by the embodiment of the present invention, the big data fusion engine is connected to the data storage end, the big data fusion engine receives the request and performs corresponding calculation processing, obtains the data storage information through the first connector of the master control node, and then obtains the data from the data source end through the second connector of the working node, so that the storage resources and the computing resources are effectively isolated, and the decoupling between storage and computing is realized. Based on this, the computing node can be statelessly expanded horizontally without being constrained and affected by the storage layer, which solves the capacity bottleneck problem of the online analysis scenario, so that both computing resources and storage resources can be fully utilized; at the same time, the connection between the computing engine and the data storage end is that the master control node connects to the master control node to obtain data storage information, and the working node connects to the working node to obtain data. The engine master control node only needs to undertake the task parsing and the request management tasks divided by the execution stage. Each engine working node directly connects to the data source working node to directly obtain data. Compared with the solution in which the engine master control node connects to the data source master node to obtain data, the data transmission path is shortened, and the workload of the engine master control node is reduced. At the same time, the data is obtained and transmitted through fragmented dispersion, which can avoid the short board effect caused by obtaining a large amount of data at one time, and can significantly improve the processing efficiency.

[0073] It should be noted that, based on the above embodiments, the embodiments of the present invention also provide corresponding improved solutions. In the preferred / improved embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be described one by one in the preferred / improved embodiments of this article.

[0074] The business party can directly connect hive to the data of the ROLAP cluster through the fusion engine to perform OLAP task analysis, but the acquisition of the data source is remote, which consumes more IO and network resources. At the same time, the update of some offline resources does not need to be real-time. In order to reduce the network interaction problem with remote storage resources, this embodiment further proposes a data caching solution based on the above embodiment. In step S104, after the engine working node receives the data of the ROLAP cluster working node through the second connector, the following steps can be further performed: determine whether to configure data cache; if so, cache the data in the memory of the engine working node.

[0075] Under the above cache configuration, before the engine master node controls the first connector to obtain the surviving computing instance in the current ROLAP cluster through the ROLAP cluster master node, the following steps need to be further performed:

[0076] (1) Determine whether the acquired data source is cache data;

[0077] (2) If yes, extract the corresponding cache data from the memory of the engine worker node and execute the step of returning the data to the downstream computing stage for computational processing;

[0078] (3) If not, execute the step of controlling the first connector through the engine master node to obtain the surviving computing instance in the current ROLAP cluster through the ROLAP cluster master node.

[0079] Change the schedule and first check whether the required underlying resources are in the cache. If so, load them directly from the cache and customize the commonly used filtering conditions related to cache data to further speed up the loading of source data.

[0080] If data cache is not configured, or data cache is not required, the processing method in this case is not limited in this embodiment, and the data can be directly calculated and processed without backup.

[0081] Set the cache on switch and configure the cache to the table. Because local memory resources are precious, before caching data to the memory of the engine worker node, you can further determine whether the number of data reads within a specified time period reaches the threshold; if it reaches the threshold, execute the step of caching the data to the memory of the engine worker node. In this method, only the data loaded into the memory is the frequently used business data to avoid excessive occupation of memory space.

[0082] If the number of reads does not reach the threshold, the present embodiment does not limit the processing method in this case, and the data may not be cached or may be stored in other remote storages.

[0083] In addition, to further ensure the smooth caching of data and avoid write failures due to insufficient space, before caching the data into the memory of the engine working node, it is possible to further determine whether the engine working node has enough memory to load the data; if so, execute the step of caching the data into the memory of the engine working node. This method can significantly improve the success rate of data caching by executing the data caching step after determining that there is enough space in the node to write.

[0084] If the engine working node does not have enough memory to load the data, this embodiment does not limit the processing method for this situation. You can re-search for a node that is sufficient to load the current data, or cache it in other spaces, such as using remote cluster cache (alluxio, a memory-based distributed file system), etc., which will not be repeated here.

[0085] Among them, judging whether the engine working node has enough memory to load the data can directly determine whether the free space is sufficient to load the data. Preferably, it can be performed according to the following steps:

[0086] Determine the free space and dirty data space in the engine worker node;

[0087] Determine whether the total amount of free space and dirty data space reaches the data loading capacity;

[0088] If it is reached, it is determined that the engine working node has enough memory to load the data; if it is not reached, it is determined that the engine working node has insufficient memory to load the data;

[0089] Accordingly, before executing the step of caching the data into the memory of the engine working node, the method further includes: clearing the dirty data of the engine working node.

[0090] After the working node of the fusion engine obtains the data split, it evaluates whether the node has enough memory to load it. The evaluation algorithm also takes into account the dirty data of the table in the working node. If there is any, the dirty data is cleared before loading. This can not only reduce the space occupied by useless data in the node, but also improve the success rate of data caching.

[0091] Furthermore, after caching the data in the memory of the engine working node, it can be determined whether the data in the ROLAP cluster is updated; if so, a data update operation request is sent to the ROLAP cluster master node.

[0092] like Figure 2 As shown, the fusion engine starts the background cache refresh task and starts 5 (configurable) worker threads for periodic scanning. If the data has been updated in the Presto cluster, it requests the Presto master node (coordinate) to update the relevant data to achieve synchronous update of the cached data.

[0093] Based on the above introduction, in the solution provided by this embodiment, if the user configures data cache, the fusion engine will cache the data related to the table into the memory of the working node of each fusion engine. When the user obtains the same data source again, it will be obtained directly from the memory, thereby reducing the IO consumption of obtaining data from the terminal.

[0094] To deepen the understanding of the above technical solution, this embodiment takes the connection to a ROLAP cluster represented by Greenplum (similar to Doris and other clusters) as an example, and the complete request processing process is as follows:

[0095] 1. After receiving the query request (SQL request) from the client, the fusion engine first distributes the request to the master node (coordinate) of Presto, performs syntax parsing and lexical parsing of the execution plan, and then breaks it down into several calculation stages (stages) and distributes them to the working nodes of the fusion engine;

[0096] 2. The working node of the fusion engine will derive tasks to process the execution plan associated with each computing stage;

[0097] 3. The task determines whether the current computing stage is the final source data end (i.e., the stage of connecting to a third-party data source). If so, the source data is requested. If not, the request is sent to the downstream computing stage.

[0098] 4. The final source data end determines the type of data source obtained. If it is hive, request the hive connector to connect to hive's distributed file system (HDFS); if it is greenplum, connect to the greenplum connector;

[0099] 5. Connectors that connect to heterogeneous data sources such as Greenplum or Hive will generate N data splits, and multiple threads will request related data, which will be cached in the worker nodes according to the engine's adaptation strategy.

[0100] Specifically, taking Greenplum as an example, the process of interacting with Greenplum data is as follows:

[0101] 1) The connector of the fusion engine first interacts with the master node of Greenplum to obtain the currently active storage data source information in Greenplum, such as address, port, library, schema, table and other related information;

[0102] The data is scattered and distributed and stored in all active nodes, so you need to find the active nodes first, then get the data from the active nodes separately, and then splice them to get the target data.

[0103] 2) The connector receives the returned data source list (list), encapsulates it in data shards, and distributes it to idle fusion engine working nodes according to the cluster task allocation strategy;

[0104] 3) When the working node of the fusion engine receives the allocation of data shards, it puts the loading tasks of the relevant data shards into the working thread and waits for queue processing;

[0105] 4) Once the working node of the fusion engine receives the throughput data from the Greenplum underlying data source, it will first adapt the Greenplum data format to the unified data format of the fusion engine, and then return it to the upper-layer computing stage for processing;

[0106] 5) If the user configures data caching, the fusion engine will cache the data related to the table in the memory of the working nodes of each fusion engine. When the user obtains the same data source of Greenplum again, it will be obtained directly from the memory, reducing the consumption of obtaining data from the terminal.

[0107] 6. The terminal's calculation stage obtains the data source and passes it to the downstream calculation stage for step-by-step processing until the final result set is returned to the presto master node (coordinate);

[0108] 7. The big data fusion engine will perform batch calculations internally and return the calculation results to the presto master node (coordinate) in batches until all result sets are returned to the client (client).

[0109] In the technical solution provided in this embodiment, the fusion engine is used to solve the massive OLAP interactive analysis scenario, meet the architecture of separation of computing and storage, so that storage resources are effectively isolated from computing resources, and computing nodes can be statelessly expanded horizontally without being constrained or affected by the storage layer, solving the capacity bottleneck problem of online analysis scenarios, so that both computing resources and storage resources can be fully utilized, and at the same time laying the foundation for the subsequent construction of a cloud-native big data analysis engine; at the same time, the fusion engine is based on the presto computing engine to connect the commonly used ROLAP and mr offline batch analysis engines. For analysis tasks with higher efficiency requirements, users can directly use the ROLAP native engine and put commonly used analysis data into ROLAP storage; for other tasks, users directly use the fusion engine to combine the commonly used business data in ROLAP and the massive data in HDFS for analysis, which effectively diverts the analysis pressure of the ROLAP cluster and improves the efficiency of online analysis tasks from data production to result output.

[0110] It should be noted that the use of the presto native engine can complete data connectivity between different heterogeneous engines. However, the native computing engine has obvious performance shortcomings in data acquisition for ROLAP-type online interactive analysis engines. In actual production use, the data sharding (split) scheduling of the data source has serious delays in some GB and TB data, which may cause business unavailability.

[0111] Corresponding to the above method embodiment, the embodiment of the present invention further provides a big data fusion engine. The big data fusion engine described below and the request response method described above can refer to each other.

[0112] like Figure 3 As shown, the big data fusion engine includes: an engine master control node; one end is connected to the master control node, and the other end is used to connect to ROLAP ( Figure 3Specifically taking Greenplum database as an example) the first connector of the master node (such as Figure 3 A bidirectional connection between the middle and upper engine master node and the greenplum master node); an engine worker node; a second connector (such as a second connector connected to the engine worker node at one end and connected to the ROLAP worker node at the other end) Figure 3 Bidirectional connection between tasks (tables) in the middle and lower engine master nodes and greenplum worker nodes).

[0113] The engine master node is used to: control the computing instances that survive in the current ROLAP cluster of the first connector; after receiving the returned data source list through the first connector, encapsulate the data source list in data shards, and distribute the data shards to idle engine working nodes according to the cluster task allocation strategy; after receiving the result set, feed back the result set to the client that initiated the ROLAP cluster query request;

[0114] The engine worker node is used to: add the loading task of the data shard to the worker thread to wait for queue processing; after receiving the data from the ROLAP cluster worker node through the second connector, return the data to the downstream computing stage for computing and processing in order to generate a result set.

[0115] The steps in the request response method described above can be implemented by the structure of the big data fusion engine. The specific implementation method can refer to the introduction of the above embodiment and will not be repeated here.

[0116] Corresponding to the above method embodiment, the embodiment of the present invention further provides a readable storage medium. The readable storage medium described below and the request response method described above can refer to each other.

[0117] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the request response method of the above method embodiment.

[0118] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0119] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. A request response method, characterized in that: include: After receiving the ROLAP cluster query request initiated by the client, the big data fusion engine controls the first connector through the engine master node to obtain the surviving computing instances in the current ROLAP cluster through the ROLAP cluster master node; After receiving the returned data source list through the first connector, the engine master control node encapsulates the data source list in data slices, and distributes the data slices to idle engine working nodes according to the cluster task allocation strategy; The engine working node adds the loading task of the data shard to the working thread to wait for queue processing; After receiving the data from the ROLAP cluster working node through the second connector, the engine working node returns the data to the downstream computing stage for computing and processing, so as to generate a result set; After receiving the result set, the engine master control node feeds the result set back to the client.

2. The request response method according to claim 1, characterized in that: Before returning the data to the downstream computing stage for computing processing, the method further includes: The data is adapted to a data format and converted into a unified data format of the big data fusion engine.

3. The request response method according to claim 1, characterized in that: After the engine working node receives the data of the ROLAP cluster working node through the second connector, the method further includes: Determine whether to configure data cache; If data caching is configured, the data is cached in the memory of the engine worker node; Accordingly, before the engine master node controls the first connector to obtain the surviving computing instance in the current ROLAP cluster through the ROLAP cluster master node, the method further includes: Determine whether the acquired data source is cached data; If so, extract the corresponding cache data from the memory of the engine working node, and execute the step of returning the data to the downstream computing stage for computing processing; If not, the step of controlling the first connector through the engine master node to obtain the surviving computing instance in the current ROLAP cluster through the ROLAP cluster master node is executed.

4. The request response method according to claim 3, characterized in that: Before caching the data in the memory of the engine working node, the method further includes: Determine whether the engine working node has sufficient memory to load the data; If yes, execute the step of caching the data into the memory of the engine working node.

5. The request response method according to claim 4, characterized in that: The determining whether the engine working node has sufficient memory to load the data includes: Determine the free space and dirty data space in the engine working node; Determine whether the total amount of the free space and the dirty data space reaches the loading amount of the data; If it is reached, it is determined that the engine working node has enough memory to load the data; if it is not reached, it is determined that the engine working node has insufficient memory to load the data; Accordingly, before executing the step of caching the data into the memory of the engine working node, the method further includes: clearing dirty data of the engine working node.

6. The request response method according to claim 3, characterized in that: After caching the data in the memory of the engine working node, the method further includes: Determine whether the data in the ROLAP cluster is updated; If so, a data update operation request is sent to the ROLAP cluster master node.

7. The request response method according to claim 3, characterized in that: Before caching the data in the memory of the engine working node, the method further includes: Determine whether the number of times the data is read within a specified time period reaches a threshold; If so, execute the step of caching the data into the memory of the engine working node.

8. A big data fusion engine, characterized in that: include: Engine master node; One end is connected to the master control node, and the other end is used to connect to the first connector of the ROLAP master control node; Engine worker nodes; One end is connected to the engine working node, and the other end is used to connect to the second connector of the ROLAP working node; The engine master control node is used to: control the computing instances that survive in the current ROLAP cluster of the first connector; after receiving the returned data source list through the first connector, encapsulate the data source list in data shards, and distribute the data shards to idle engine working nodes according to the cluster task allocation strategy; after receiving the result set, feed back the result set to the client that initiated the ROLAP cluster query request; The engine working node is used to: add the loading task of the data shard to the working thread to wait for queue processing; After receiving the data from the working node of the ROLAP cluster through the second connector, the data is returned to the downstream computing stage for computational processing to generate a result set.

9. A request response system, characterized in that: include: The big data fusion engine and ROLAP cluster as claimed in claim 8; The first connector of the big data fusion engine is connected to the master node of the ROLAP cluster, and the second connector of the big data fusion engine is connected to the working node of the ROLAP cluster.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the request response method according to any one of claims 1 to 7 are implemented.