Data processing method, apparatus and electronic device

Abstract

Embodiments of the present application disclose a data processing method, device and electronic equipment. A specific implementation of the method comprises: receiving a data query request for a first data processing engine; in a front-end node, selecting a target data processing engine from a data processing engine candidate set based on the data query request; if the target data processing engine is the first data processing engine, parsing the data query request in the front-end node, sending the parsed result to a back-end node to enable the back-end node to perform data query; and if the target data processing engine is a second data processing engine, in the front-end node, generating target information based on the data query request and sending the target information to the second data processing engine to enable the second data processing engine to acquire data from the first data processing engine to perform data query. The implementation can use a data processing engine with higher processing capability to perform data analysis on data stored in the data processing engine, meeting the needs of lake-warehouse integration.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to data processing methods, apparatus, and electronic devices. Background Technology

[0002] OLAP (Online Analytical Processing) systems are a primary application of data warehouse systems. They support complex analytical operations, focusing on decision support for decision-makers and senior management. They can quickly and flexibly process complex queries on large datasets according to analysts' requirements and provide the results to decision-makers. However, when the data to be processed reaches a certain scale (e.g., petabytes), existing OLAP systems are unable to handle it due to insufficient memory and long processing times. Therefore, addressing the problem of existing OLAP systems' inability to process large datasets is urgently needed. Summary of the Invention

[0003] This disclosure is provided to briefly introduce the concepts, which will be described in detail in the subsequent detailed description section. This disclosure is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, embodiments of this disclosure provide a data processing method, comprising: receiving a data query request for a first data processing engine, wherein the first data processing engine includes a front-end node and a back-end node, the front-end node receiving the data query request, and the back-end node storing data; in the front-end node, selecting a target data processing engine from a candidate set of data processing engines based on the data query request, wherein the target data processing engine processes the data query request, the candidate set of data processing engines includes a first data processing engine and a second data processing engine, the second data processing engine having a higher data processing capability than the first data processing engine; if the target data processing engine is the first data processing engine, then in the front-end node, parsing the data query request and sending the parsing result to the back-end node so that the back-end node can perform a data query; if the target data processing engine is the second data processing engine, then in the front-end node, generating target information based on the data query request and sending it to the second data processing engine so that the second data processing engine can obtain data from the first data processing engine for a data query.

[0005] Secondly, embodiments of this disclosure provide a data processing apparatus, comprising: a receiving unit, configured to receive a data query request for a first data processing engine, wherein the first data processing engine includes a front-end node and a back-end node, the front-end node receiving the data query request and the back-end node storing data; a selection unit, configured to select a target data processing engine from a candidate set of data processing engines in the front-end node based on the data query request, wherein the target data processing engine processes the data query request, the candidate set of data processing engines includes a first data processing engine and a second data processing engine, the second data processing engine having a higher data processing capability than the first data processing engine; a first processing unit, configured to, if the target data processing engine is the first data processing engine, parse the data query request in the front-end node and send the parsing result to the back-end node so that the back-end node can perform a data query; and a second processing unit, configured to, if the target data processing engine is the second data processing engine, generate target information in the front-end node based on the data query request and send it to the second data processing engine so that the second data processing engine can obtain data from the first data processing engine for a data query.

[0006] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in the first aspect.

[0007] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the steps of the data processing method as described in the first aspect.

[0008] The data processing method, apparatus, and electronic device provided in this disclosure receive a data query request for a first data processing engine. Then, in the front-end node of the first data processing engine, based on the data query request, it selects from a candidate set of data processing engines whether to process the data query request using either the first data processing engine itself or a second data processing engine with higher processing capabilities. If the first data processing engine is selected, the front-end node parses the data query request and sends the parsing result to the back-end node for data querying. If the second data processing engine is selected, the front-end node generates target information based on the data query request and sends it to the second data processing engine, allowing the second data processing engine to retrieve data from the first data processing engine for data querying. This approach allows the first data processing engine to perform data analysis on the data stored in the first data processing engine when it is unable to handle a larger number of received data query requests, thus meeting the requirements of a lake warehouse integration. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a flowchart of an embodiment of the data processing method according to the present disclosure;

[0011] Figure 2 This is a schematic diagram of a data processing method according to the present disclosure;

[0012] Figure 3 This is a schematic diagram of a data retrieval method based on the data processing method disclosed herein;

[0013] Figure 4 This is a flowchart of an embodiment selected by the data processing engine in the data processing method of this disclosure;

[0014] Figure 5 This is a schematic diagram of the structure of an embodiment of the data processing apparatus according to the present disclosure;

[0015] Figure 6 These are exemplary system architecture diagrams to which the various embodiments of this disclosure can be applied;

[0016] Figure 7This is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0019] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] Please refer to Figure 1 The diagram illustrates a flow 100 of an embodiment of a data processing method according to the present disclosure. This data processing method includes the following steps:

[0024] Step 101: Receive a data query request for the first data processing engine.

[0025] In this embodiment, the execution entity of the data processing method can receive data query requests for the first data processing engine. These data query requests are typically SQL (Structured Query Language) statements. SQL is a database language with multiple functions, including data manipulation and definition. This language is interactive and provides great convenience to users. Database management systems should fully utilize SQL to improve the working quality and efficiency of computer application systems.

[0026] Lakewarehousing is a new type of open architecture that connects data warehouses and data lakes, combining the high performance and management capabilities of data warehouses with the flexibility of data lakes. The underlying layer supports the coexistence of multiple data types and enables data sharing between them. The upper layer can access the data through a unified encapsulated interface, and can simultaneously support real-time querying and analysis, bringing more convenience to enterprises for data governance.

[0027] Here, the aforementioned first data processing engine typically includes front-end nodes (FE) and back-end nodes (BE). The front-end nodes are typically used to receive data query requests. In addition, they can manage metadata, manage client connections, perform query parsing and planning, generate query execution plans, and schedule queries (distributing queries to BEs for execution). The data in the first data processing engine is typically stored in the back-end nodes. Furthermore, the back-end nodes are used for executing query execution plans and managing replicas. In other words, the aforementioned first data processing engine is typically the data warehouse within a lakeware repository.

[0028] Step 102: In the front-end node, select the target data processing engine from the candidate set of data processing engines based on the data query request.

[0029] In this embodiment, the execution entity can select a target data processing engine from the candidate set of data processing engines based on the data query request in the front-end node of the first data processing engine. The target data processing engine is typically the data processing engine that processes the data query request. The candidate set of data processing engines may include the first data processing engine and a second data processing engine, with the second data processing engine typically having a higher data processing capability than the first data processing engine.

[0030] Here, the aforementioned second data processing engine is typically the data lake within a lake warehouse system.

[0031] As an example, the first data processing engine described above can have data analysis capabilities at the TB (terabyte) level, while the second data processing engine described above can have data analysis capabilities at the PB level and above. Here, 1 PB = 1024 TB.

[0032] Here, the aforementioned front-end node can analyze the data query request to determine whether its computational logic is simple or complex. If it is determined to be simple logic, the first data processing engine can be selected as the target data processing engine. If it is determined to be complex logic, the second data processing engine can be selected as the target data processing engine.

[0033] Step 103: If the target data processing engine is the first data processing engine, then in the front-end node, the data query request is parsed and the parsing result is sent to the back-end node so that the back-end node can perform data query.

[0034] In this embodiment, if the selected target data processing engine is the first data processing engine, the execution entity can parse the data query request in the front-end node and send the parsing result to the back-end node so that the back-end node can perform data query.

[0035] Specifically, the aforementioned front-end node can parse the aforementioned data query request, determine the storage location of the data queried in the data table of the aforementioned back-end node, and generate a data query task. It can then send the data storage location and the data query task to the aforementioned back-end node. Upon receiving the data storage location and the data query task, the aforementioned back-end node can execute the data query task, retrieving data from the corresponding storage location for data querying and analysis.

[0036] Step 104: If the target data processing engine is the second data processing engine, then in the front-end node, based on the data query request, target information is generated and sent to the second data processing engine so that the second data processing engine can obtain data from the first data processing engine for data query.

[0037] In this embodiment, if the selected target data processing engine is the second data processing engine, the execution entity can generate target information and send it to the second data processing engine in the front-end node based on the data query request, so that the second data processing engine can obtain data from the first data processing engine for data query.

[0038] Specifically, the aforementioned front-end node can directly send the aforementioned data query request as target information to the aforementioned second data processing engine. After receiving the aforementioned data query request, the aforementioned second data processing engine can parse the aforementioned data query request and use the parsing result to obtain the corresponding data from the aforementioned first data processing engine for data querying.

[0039] As an example, the above parsing results may include the storage location of the data queried by the above data query request in the backend node of the above first data processing engine.

[0040] The method provided in the above embodiments of this disclosure involves receiving a data query request for a first data processing engine; then, in the front-end node of the first data processing engine, based on the data query request, selecting from a candidate set of data processing engines whether to process the data for the data query request using either the first data processing engine itself or a second data processing engine with higher processing capabilities; if the first data processing engine is selected, the front-end node parses the data query request and sends the parsing result to the back-end node for data querying; if the second data processing engine is selected, the front-end node generates target information based on the data query request and sends it to the second data processing engine for data querying, allowing the second data processing engine to retrieve data from the first data processing engine. This approach allows the first data processing engine to perform data analysis on the data stored in the first data processing engine when it is unable to analyze a larger number of received data query requests, thus meeting the requirements of lake warehouse integration.

[0041] In some optional implementations, the aforementioned execution entity can further select a target data processing engine from the candidate set of data processing engines in the aforementioned front-end node based on the aforementioned data query request, in the following manner: In the aforementioned front-end node, the storage information corresponding to the data queried by the aforementioned data query request can be determined. The aforementioned storage information may include at least one of the following: the space of the storage table, the number of partitions, and the number of files. The space of the storage table typically refers to the space of the storage table where the data queried by the aforementioned data query request resides. The number of partitions typically refers to the number of regions from which the data queried by the aforementioned data query request originates. The number of files typically refers to the number of files from which the data queried by the aforementioned data query request originates.

[0042] Subsequently, the aforementioned front-end node can compare the space of the aforementioned storage table with at least one of the following: the number of partitions with at least one of the following: the number of partitions with at least one of the following: the number of files with at least one of the following: the space of the aforementioned storage table is greater than the preset space threshold; the number of partitions is greater than the preset partition threshold; and the number of files is greater than the preset file threshold. If at least one of these conditions is met, then the second data processing engine can be selected to process the aforementioned data query request. This method selects a data processing engine based on the space occupied by the data queried in the aforementioned data query request, thereby selecting a more suitable data processing engine to process the data.

[0043] In some alternative implementations, the aforementioned execution entity can further select a target data processing engine from the candidate set of data processing engines in the aforementioned front-end node based on the aforementioned data query request, in the following manner: In the aforementioned front-end node, the predicted resource consumption during the processing of the aforementioned data query request can be determined. Resources may include, but are not limited to, at least one of the following: CPU (Central Processing Unit), memory, network, and I / O (Input / Output).

[0044] Subsequently, the aforementioned front-end node can compare the predicted resource consumption value with a preset resource consumption threshold. If it is determined that the predicted resource consumption value is greater than the preset resource consumption threshold, the execution entity can select a second data processing engine to process the data query request. This method selects a data processing engine based on the resource consumption of the data queried in the data query request during processing, thereby selecting a more suitable data processing engine to process the data.

[0045] In some optional implementations, if a second data processing engine is selected to process the data query request, the execution entity can generate target information based on the data query request and send it to the second data processing engine in the front-end node in the following way: the data query request can be parsed in the front-end node to obtain a first dataset, which can be processed by the first data processing engine.

[0046] Subsequently, the aforementioned front-end node can determine whether the first dataset can be converted into a second dataset, which can then be processed by the second data processing engine. The front-end node can store a mapping table showing the correspondence between the first and second datasets, and can use this mapping table to determine whether the first dataset can be converted into the second dataset.

[0047] If it is determined that the first dataset can be transformed into the second dataset, the aforementioned front-end node can use the aforementioned correspondence table to transform the first dataset into the second dataset. Then, the second dataset can be sent as target information to the second data processing engine. This approach allows the second data processing engine to further analyze the parsing results from the first data processing engine.

[0048] As an example, if the first data processing engine is StarRocks and the second data processing engine is Spark, then the SQL statement needs to be parsed into a task that the Spark engine can recognize on the FE side of the StarRocks engine. After the FE side parses the SQL statement, it obtains an executable physical execution plan. However, the Spark engine cannot recognize this physical execution plan and needs to transform it according to certain rules.

[0049] For example, the Scan Operator in the StarRocks engine for querying can be converted into an RDD related to readFile in the Spark engine; the Exchange Operator in the StarRocks engine can be converted into an RDD related to Reduce in Spark; the Aggregate Operator in the StarRocks engine can be converted into an RDD related to join in Spark; and the Sort Operator in the StarRocks engine can be converted into an RDD related to sort in Spark.

[0050] In some alternative implementations, after determining whether the first dataset can be transformed into the second dataset, if it is determined that the first dataset cannot be transformed into the second dataset, the front-end node can send the data query request as the target information to the second data processing engine. In other words, for operators not covered, the FE will directly send the SQL statement to the Spark engine.

[0051] In some optional implementations, if the first data processing engine is selected to process the aforementioned data query request, the execution entity can parse the data query request in the front-end node and send the parsing result to the back-end node for data querying: In the front-end node, the data query request can be parsed to generate a physical execution plan. The physical execution plan is an execution tree composed of physical operators. The SQL statement is processed through stages such as parsing and analyzing to finally generate the physical execution plan. Then, the physical execution plan can be split into multiple plan fragments, and fragment instances can be created based on these fragments, which are then sent to the back-end node.

[0052] Here, PlanFragment is a part of the physical execution plan. Only when the physical execution plan is split into several PlanFragments by the FE can it be executed in parallel on multiple machines. PlanFragment is also composed of physical operators, and also contains DataSink. The upstream PlanFragment sends data to the Exchange operator of the downstream PlanFragment through the DataSink.

[0053] A Fragment Instance is an execution instance of a PlanFragment. StarRocks tables are partitioned and bucketed into several tablets, each of which is stored as multiple replicas on compute nodes. PlanFragments can be instantiated into multiple Fragment Instances to process tablets distributed across different machines, thus achieving parallel data computation. The FE determines the number of Fragment Instances and the target BE (Executable Entity) to execute the Fragment Instances, and then the FE delivers the Fragment Instances to the BE.

[0054] In the aforementioned backend nodes, a pipeline can be used to process the fragmented instances for data querying, thus providing a way for a primary data processing engine to process data query requests. A pipeline is a chain of operators. The SourceOperator, as the starting operator of the pipeline, generates data for subsequent operators, while the SinkOperator, as the ending operator, absorbs the calculation results and outputs the data.

[0055] In the Pipeline execution engine, the PipelineBuilder on the BE will further split the PlanFragment into several Pipelines. Each Pipeline will be instantiated into a set of PipelineDrivers according to the Pipeline parallelism parameter. The PipelineDriver is a Pipeline instance and also the basic task that the Pipeline execution engine can schedule.

[0056] In some optional implementations, the aforementioned first data processing engine can include a Massively Parallel Processing (MPP) database. As an example, this first database could be StarRocks, a next-generation, high-speed, full-scenario MPP database. StarRocks' vision is to make data analysis simpler and more agile for users. Users can use StarRocks to support rapid analysis across various data analysis scenarios without complex preprocessing. StarRocks has a simple architecture, employs a fully vectorized engine, and is equipped with a newly designed CBO (Cost Based Optimizer), resulting in query speeds (especially for multi-table joins) far exceeding those of similar products. StarRocks effectively supports real-time data analysis and enables efficient querying of real-time updated data. StarRocks also supports modern materialized views, further accelerating queries. Using StarRocks, users can flexibly build various models, including wide tables, star schemas, and snowflake schemas. StarRocks is compatible with the MySQL (open-source relational database management system) protocol, supports standard SQL syntax, is easy to integrate, has no external dependencies, is highly available, and is easy to operate and manage.

[0057] The aforementioned second data processing engine can include a distributed computing framework. As an example, this second data processing engine could be Spark, an open-source distributed computing framework designed specifically for large-scale data analysis and processing. It utilizes in-memory computing technology and a Directed Acyclic Graph (DAG) to provide faster analytical processing capabilities than the MapReduce engine. Spark also boasts petabyte-scale (PB) and higher offline analysis capabilities.

[0058] Because StarRocks uses an MPP architecture, it is suitable for real-time data warehouse scenarios. However, when the data volume reaches the petabyte (PB) level, StarRocks is generally unsuitable for data processing due to insufficient memory, long execution times, and the lack of a retry mechanism when tasks fail. Spark can solve these problems. Therefore, combining StarRocks and Spark can satisfy both real-time requirements (real-time analysis capabilities for TB-level data) and large data volume requirements (offline analysis capabilities for PB-level and above data).

[0059] like Figure 2 As shown, Figure 2 A schematic diagram 200 illustrates one processing method of the data processing approach. In Figure 2 In StarRocks, a cluster typically consists of FrontEnds (FEs) and numerous BackEnds (BEs). FEs are the front-end nodes of StarRocks, primarily responsible for cluster metadata management, client connection management, query parsing and planning, query execution plan generation, and query scheduling (distributing queries to BEs for execution). BEs are the back-end nodes of StarRocks, primarily responsible for data storage, query execution plan execution, and replica management. Data in each table is distributed across multiple Tablets using partitioning or bucketing mechanisms. To ensure fault tolerance, each Tablet is replicated, and these Tablets are ultimately distributed across different BEs.

[0060] SparkContext is the entry point for the Spark computing framework, responsible for managing Spark distributed resources, creating RDDs (Resilient Distributed Datasets), and scheduling tasks. SparkSession is the entry point for SparkSQL, responsible for parsing, analyzing, and optimizing SQL, generating physical plans, and scheduling SQL tasks. Driver is the process that hosts SparkContext in the Spark distributed processing framework; it is responsible for running SparkContext, scheduling and managing Executors. There is only one Driver. Executors are the processes that execute distributed tasks, responsible for executing the tasks distributed by the Driver. There are multiple Executors.

[0061] SQL statements enter the StarRocks Front End (FE) through a unified SQL entry point. The FE parses the SQL statement and determines whether to use the StarRocks engine for real-time data analysis or the Spark engine for discrete data analysis. If the StarRocks engine is used, the FE sends the execution task to the Back End (BE) for data analysis. If the Spark engine is used, the FE sends the execution task to the Driver. The Driver schedules multiple Executor processes, and the Executors query the data stored on the BE.

[0062] In some optional implementations, the data query request may include an engine identifier specifying a data processing engine. Users can automatically analyze the characteristics of the SQL statement, manually specify the data processing engine to be processed, and add the engine identifier to the data query request. The execution entity can further select a target data processing engine from the candidate set of data processing engines in the aforementioned front-end node based on the data query request in the following way: The execution entity can select the data processing engine specified in the data query request from the candidate set of data processing engines and perform data processing on the data query request. This approach allows the selection of the data processing engine to better meet user needs.

[0063] It should be noted that if the user specifies the StarRocks engine, the SQL statement must conform to StarRocks syntax; if the user specifies the Spark engine, SparkSQL is responsible for parsing and executing the SQL statement, therefore, the SQL statement must conform to SparkSQL syntax.

[0064] In some optional implementations, if a second data processing engine is selected to query the data request, the front-end node can further generate target information based on the data query request and send it to the second data processing engine in the following way: the front-end node can send the data query request as target information to the second data processing engine. That is, in scenarios where the user manually specifies the data processing engine, if a second data processing engine is specified, the second data processing engine is responsible for parsing and executing the SQL statement.

[0065] In some optional implementations, the target information sent to the second data processing engine is the data query request. After generating the target information based on the data query request and sending it to the second data processing engine, the front-end node can determine whether the second data processing engine caches metadata. This metadata refers to the metadata of the data table storing the data queried by the data query request. Table metadata refers to data about the table data, from table definitions and field attributes to table relationships and permissions. Additional metadata includes data analysis and statistical information, such as indexes and partitions.

[0066] If it is determined that the second data processing engine does not cache metadata, then the metadata can be obtained from the front-end node and cached in the memory of the second data processing engine. Within the second data processing engine, this metadata can be used to retrieve data from the first data processing engine for data querying. This method satisfies the need for the second data processing engine to query data from the first data processing engine.

[0067] In some alternative implementations, when the dataset transformation process fails to convert it into operators that the Spark engine can handle, or in scenarios where the Spark engine is manually specified for data analysis, SparkSQL is responsible for parsing and executing the SQL statement. In this case, SparkSQL needs to know the metadata of the table in the SQL statement. FE (Front-End Node) is the front-end node of StarRocks, responsible for managing metadata, managing client connections, query planning, query scheduling, and other tasks. Each FE node maintains a complete copy of the metadata in memory, allowing each FE node to provide consistent service. Spark can send requests for metadata to the FE via the HTTP (Hypertext Transfer Protocol) engine and cache it in memory. The next time the metadata information is read, it checks if it's cached in memory; if so, it reads it directly from memory; otherwise, it sends the request to the FE again.

[0068] In some alternative implementations, after determining whether metadata is cached in the second data processing engine, if it is determined that metadata is cached in the second data processing engine, data can be retrieved from the first data processing engine using the cached metadata for data querying. This method allows direct use of locally stored metadata to retrieve data from the first data processing engine, avoiding the need to retrieve metadata before each data processing step, thus improving data processing efficiency and reducing resource consumption.

[0069] In some optional implementations, the aforementioned execution entity can retrieve data from the aforementioned first data processing engine within the second data processing engine using the aforementioned metadata for data querying: Within the second data processing engine, a data retrieval request can be sent to the aforementioned front-end node using the aforementioned metadata, and the data received from the aforementioned front-end node can be used for data querying. The aforementioned data retrieval request is used to request the data queried in the aforementioned data query request. In this way, the second data processing engine can retrieve data from the first data processing engine, thereby performing corresponding data queries and data analysis.

[0070] In some alternative implementations, the aforementioned execution entity can retrieve data from the aforementioned first data processing engine within the second data processing engine using the aforementioned metadata for data querying: The second data processing engine can directly pull data from the aforementioned backend nodes using the aforementioned metadata, thereby performing data queries on the pulled data. This approach allows the second data processing engine (e.g., Spark) to directly pull data from the data storage location (i.e., the backend node), instead of having it forwarded by the frontend node, thus improving the efficiency of the second data processing engine in reading and writing data from the first data processing engine.

[0071] In some optional implementations, the data in the backend nodes is stored in the form of data shards (Tablets), and the metadata includes the metadata of the data shards, such as the index and location of the data shards.

[0072] The aforementioned execution entity can utilize the aforementioned metadata to pull data from the aforementioned backend nodes in the following manner: The aforementioned second data processing engine can generate at least one data pull task according to data shards. Here, a data pull task can pull data corresponding to at least one data shard. For example, a data pull task can pull data corresponding to three data shards. Then, the aforementioned at least one data pull task can be executed to pull data from the aforementioned backend nodes.

[0073] Here, the data retrieval task can be divided according to the number of data shards contained in a BE end, that is, an Executor process can retrieve data from a data shard on a BE end.

[0074] like Figure 3 As shown, Figure 3 A schematic diagram 300 illustrates a data retrieval method for data processing. In Figure 3 In StarRocks, data is typically stored in units of Tablets, which are the smallest unit of data management in StarRocks.

[0075] Spark divides tasks and retrieves data at the tablet level. The Spark Driver obtains metadata information from the FE (Feature Entity), such as a list of tablets and corresponding information, divides tasks, and distributes the tasks to the Executors. The Executors directly pull data from the physical storage location of the tablets on the BE (Body Entity) for computation.

[0076] This approach allows Spark to pull data directly from the data storage location (i.e., the BE end) instead of having it forwarded by the FE end, thereby improving the efficiency of Spark engine reading and writing data in the StarRocks engine.

[0077] In some optional implementations, the front-end node can obtain the processing results for the data query requests from the second data processing engine. Since the Spark engine tasks are initiated by the StarRocks engine's Front End (FE), and the FE also needs to monitor the task execution status, it periodically sends task execution status requests to the Spark engine's Driver. In this way, the first data processing engine can monitor the task execution status of the second data processing engine.

[0078] Further reference Figure 4 This illustrates a flow 400 of an embodiment of a data processing engine selected in a data processing method. The flow 400 selected by the data processing engine includes the following steps:

[0079] Step 401: Receive a data query request for the first data processing engine.

[0080] In this embodiment, step 401 can be performed in a similar manner to step 101, and will not be described again here.

[0081] Step 402: In the front-end node, determine the storage information corresponding to the data queried by the data query request.

[0082] In this embodiment, the execution entity of the data processing method can determine the storage information corresponding to the data queried by the data query request in the front-end node. The storage information may include at least one of the following: the space of the storage table, the number of partitions, and the number of files. The space of the storage table typically refers to the space of the storage table where the data queried by the data query request resides. The number of partitions typically refers to the number of regions from which the data queried by the data query request originates. The number of files typically refers to the number of files from which the data queried by the data query request originates.

[0083] Step 403: Determine whether at least one of the following conditions is met: the storage table space is greater than the preset space threshold, the number of partitions is greater than the preset partition number threshold, or the number of files is greater than the preset file number threshold.

[0084] In this embodiment, the execution entity can determine whether the space of the storage table is greater than a preset space threshold, whether the number of partitions is greater than a preset partition number threshold, and whether the number of files is greater than a preset file number threshold.

[0085] If it is determined that the space of the above storage table is greater than the preset space threshold, the number of the above partitions is greater than the preset partition number threshold, and the number of the above files is greater than the preset file number threshold, then the above execution entity can execute step 404.

[0086] If it is determined that the space of the above storage table is greater than the preset space threshold, the number of the above partitions is greater than the preset partition number threshold, and the number of the above files is greater than the preset file number threshold, then the above execution entity can execute step 406.

[0087] Step 404: If it is determined that the storage table space is greater than the preset space threshold, the number of partitions is greater than the preset partition number threshold, and the number of files is greater than the preset file number threshold, then in the front-end node, the predicted value of resource consumption during the processing of data query requests is determined.

[0088] In this embodiment, if it is determined in step 403 that the following conditions are not met: the space of the storage table is greater than a preset space threshold, the number of partitions is greater than a preset partition number threshold, and the number of files is greater than a preset file number threshold, then the predicted resource consumption during the data query request processing process can be determined in the aforementioned front-end node. Resources may include, but are not limited to, at least one of the following: CPU, memory, network, and I / O.

[0089] Step 405: Determine whether the predicted resource consumption value is greater than the preset resource consumption threshold.

[0090] In this embodiment, the execution entity can determine whether the predicted resource consumption value is greater than a preset resource consumption threshold. The execution entity can compare the predicted resource consumption value with the preset resource consumption threshold. If it is determined that the predicted resource consumption value is greater than the preset resource consumption threshold, the execution entity can execute step 406; if it is determined that the predicted resource consumption value is less than or equal to the preset resource consumption threshold, the execution entity can execute step 407.

[0091] Step 406: If it is determined that the storage table space is greater than a preset space threshold, the number of partitions is greater than a preset partition number threshold, or the number of files is greater than a preset file number threshold, or if it is determined that the predicted resource consumption value is greater than a preset resource consumption threshold, then the second data processing engine is selected to process the data query request.

[0092] In this embodiment, if it is determined in step 403 that the storage table space is greater than a preset space threshold, the number of partitions is greater than a preset partition number threshold, or the number of files is greater than a preset file number threshold, or if it is determined in step 405 that the predicted resource consumption value is greater than a preset resource consumption threshold, then the execution entity can select a second data processing engine with stronger processing capabilities to process the data query request.

[0093] Step 407: If the predicted resource consumption value is determined to be less than or equal to the preset resource consumption threshold, then the first data processing engine is selected to process the data query request.

[0094] In this embodiment, if the predicted resource consumption value is determined to be less than or equal to the preset resource consumption threshold in step 405, the execution entity can select the first data processing engine to process the data query request.

[0095] from Figure 4 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the data processing method flow 400 in this embodiment embodies the step of selecting a first data processing engine or a second data processing engine to process the data query request by utilizing the storage information corresponding to the data queried in the data query request and the predicted resource consumption value for processing the data query request. Therefore, the solution described in this embodiment can more rationally select the appropriate data processing engine for data processing.

[0096] Further reference Figure 5 As an implementation of the methods shown in the above figures, this application provides an embodiment of a data processing apparatus, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0097] like Figure 5As shown, the data processing device 500 of this embodiment includes: a receiving unit 501, a selection unit 502, a first processing unit 503, and a second processing unit 504. The receiving unit 501 is used to receive data query requests for a first data processing engine, wherein the first data processing engine includes a front-end node and a back-end node, the front-end node receives the data query requests, and the back-end node stores the data; the selection unit 502 is used in the front-end node to select a target data processing engine from a candidate set of data processing engines based on the data query requests, wherein the target data processing engine processes the data query requests, and the candidate set of data processing engines includes a first data processing engine and a second data processing engine, the second data processing engine having a higher data processing capability than the first data processing engine; the first processing unit 503 is used to parse the data query requests in the front-end node if the target data processing engine is the first data processing engine, and send the parsing results to the back-end node so that the back-end node can perform data queries; the second processing unit 504 is used to generate target information in the front-end node based on the data query requests and send it to the second data processing engine if the target data processing engine is the second data processing engine, so that the second data processing engine can obtain data from the first data processing engine for data queries.

[0098] In this embodiment, the specific processing of the receiving unit 501, the selection unit 502, the first processing unit 503, and the second processing unit 504 of the data processing device 500 can be referred to Figure 1 The corresponding steps are 101, 102, 103 and 104 in the embodiment.

[0099] In some optional implementations, the selection unit 502 can be further used to select a target data processing engine from a candidate set of data processing engines in the front-end node based on a data query request in the following manner: in the front-end node, determining the storage information corresponding to the data queried by the data query request, wherein the storage information includes at least one of the following: the space of the storage table, the number of partitions, and the number of files; in response to determining at least one of the following, selecting a second data processing engine as the target data processing engine: the space of the storage table is greater than a preset space threshold, the number of partitions is greater than a preset partition number threshold, and the number of files is greater than a preset file number threshold.

[0100] In some alternative implementations, the selection unit 502 may be further used to select a target data processing engine from a candidate set of data processing engines in the front-end node based on a data query request in the following manner: in the front-end node, determining the predicted value of resource consumption during the processing of the data query request; in response to determining that the predicted value of resource consumption is greater than a preset resource consumption threshold, selecting a second data processing engine as the target data processing engine.

[0101] In some optional implementations, the second processing unit 504 can be further used to generate target information and send it to the second data processing engine in the front-end node based on the data query request in the following manner: in the front-end node, the data query request is parsed to obtain a first dataset, wherein the first dataset can be processed by the first data processing engine; it is determined whether the first dataset can be converted into a second dataset, wherein the second dataset can be processed by the second data processing engine; if so, the first dataset is converted into a second dataset, and the second dataset is sent to the second data processing engine as target information.

[0102] In some alternative implementations, the second processing unit 504 may be further configured to send a data query request as target information to a second data processing engine if it is determined that the first dataset cannot be converted into the second dataset.

[0103] In some optional implementations, the data query request may include an engine identifier for a specified data processing engine; and the selection unit 502 may be further configured to select a target data processing engine from a candidate set of data processing engines in the front-end node based on the data query request in the following manner: selecting the data processing engine specified in the data query request as the target data processing engine from the candidate set of data processing engines.

[0104] In some alternative implementations, the second processing unit 504 can be further used to generate target information and send it to the second data processing engine in the front-end node based on the data query request in the following manner: in the front-end node, the data query request is sent as target information to the second data processing engine.

[0105] In some optional implementations, the target information is a data query request; and the aforementioned data processing apparatus 500 may include a determining unit (not shown in the figure) and a caching unit (not shown in the figure). The determining unit is used to determine whether metadata is cached in the second data processing engine, wherein the metadata is the metadata of the data table storing the data queried by the data query request; the caching unit is used to obtain the metadata from the front-end node if the second data processing engine does not cache the metadata, cache the metadata in the memory of the second data processing engine, and in the second data processing engine, use the metadata to obtain data from the first data processing engine for data querying.

[0106] In some alternative implementations, the data processing apparatus 500 may include a first acquisition unit (not shown in the figure). This acquisition unit is used to retrieve data from the first data processing engine using the metadata cached in the second data processing engine for data querying.

[0107] In some optional implementations, the aforementioned caching unit or the aforementioned first acquisition unit can be further used in the second data processing engine to acquire data from the first data processing engine using metadata for data querying in the second data processing engine as follows: in the second data processing engine, a data acquisition request is sent to the front-end node using metadata, and the data fed back by the front-end node is received for data querying.

[0108] In some optional implementations, the aforementioned caching unit or the aforementioned first acquisition unit can be further used to retrieve data from the first data processing engine using metadata in the second data processing engine for data querying in the following manner: in the second data processing engine, data is pulled from the backend node using metadata, and data querying is performed on the pulled data.

[0109] In some optional implementations, data in the backend nodes is stored according to data shards, and the metadata includes the metadata of the data shards; and data can be pulled from the backend nodes by utilizing the metadata in the following ways: generating at least one data pull task according to the data shards, wherein a data pull task pulls data corresponding to at least one data shard; and executing at least one data pull task to pull data from the backend nodes.

[0110] In some optional implementations, the first processing unit 503 described above can be further used to parse the data query request in the front-end node and send the parsing result to the back-end node so that the back-end node can perform data query: in the front-end node, the data query request is parsed, a physical execution plan is generated, the physical execution plan is split into multiple plan fragments, fragment instances are created based on the multiple plan fragments, and the fragment instances are sent to the back-end node; in the back-end node, the fragment instances are processed using pipeline links to query the data.

[0111] In some alternative implementations, the data processing apparatus 500 may include a second acquisition unit (not shown in the figure). The second acquisition unit may be used to acquire the processing results for the data query request from the second data processing engine in the front-end node.

[0112] In some alternative implementations, the first data processing engine mentioned above includes a massively parallel processing database, and the second data processing engine mentioned above includes a distributed computing framework.

[0113] Figure 6 An exemplary system architecture 600 is shown, to which embodiments of the data processing methods of this disclosure can be applied.

[0114] like Figure 6 As shown, system architecture 600 may include terminal devices 6011, 6012, and 6013, network 602, and server 603. Network 602 is used as a medium to provide communication links between terminal devices 6011, 6012, and 6013 and server 603. Network 602 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0115] Users can use terminal devices 6011, 6012, and 6013 to interact with server 603 via network 602 to send or receive messages, etc. For example, server 603 can receive data query requests for the first data processing engine sent by terminal devices 6011, 6012, and 6013. Various communication client applications, such as short video software and search engines, can be installed on terminal devices 6011, 6012, and 6013.

[0116] Terminal devices 6011, 6012, and 6013 can be either hardware or software. When terminal devices 6011, 6012, and 6013 are hardware, they can be various electronic devices with displays and supporting information interaction, including but not limited to smartphones, tablets, and laptops. When terminal devices 6011, 6012, and 6013 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0117] Server 603 can be a server providing various services. For example, it can be a backend server that processes data query requests. Server 603 can receive data query requests for a first data processing engine, wherein the first data processing engine includes a frontend node and a backend node, the frontend node receives the data query request, and the backend node stores the data; then, in the frontend node, based on the data query request, a target data processing engine can be selected from a candidate set of data processing engines, wherein the target data processing engine processes the data query request, and the candidate set of data processing engines includes the first data processing engine and a second data processing engine, wherein the data processing capability of the second data processing engine is higher than that of the first data processing engine; if the target data processing engine is the first data processing engine, then in the frontend node, the data query request is parsed, and the parsing result is sent to the backend node so that the backend node can perform a data query; if the target data processing engine is the second data processing engine, then in the frontend node, based on the data query request, target information is generated and sent to the second data processing engine so that the second data processing engine can obtain data from the first data processing engine for a data query.

[0118] It should be noted that server 603 can be either hardware or software. When server 603 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 603 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0119] It should also be noted that the data processing method provided in this embodiment is usually executed by server 603, and the data processing device is usually located in server 603.

[0120] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0121] The following is for reference. Figure 7 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 6 The structural diagram of the server (700) in the middle. Figure 7 The server shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0122] like Figure 7As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0123] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0124] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by a processing device 701, it performs the functions defined in the methods of embodiments of this disclosure. It should be noted that the computer-readable medium described in embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0125] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following: It receives a data query request for a first data processing engine, wherein the first data processing engine includes a front-end node and a back-end node, the front-end node receives the data query request, and the back-end node stores data; in the front-end node, based on the data query request, it selects a target data processing engine from a candidate set of data processing engines, wherein the target data processing engine processes the data query request, and the candidate set of data processing engines includes the first data processing engine and a second data processing engine, the second data processing engine having a higher data processing capability than the first data processing engine; if the target data processing engine is the first data processing engine, in the front-end node, it parses the data query request and sends the parsing result to the back-end node so that the back-end node can perform a data query; if the target data processing engine is the second data processing engine, in the front-end node, based on the data query request, it generates target information and sends it to the second data processing engine so that the second data processing engine can obtain data from the first data processing engine for a data query.

[0126] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be located in a processor; for example, a processor may be described as including a receiving unit, a selecting unit, a first processing unit, and a second processing unit. The names of these units do not necessarily limit the specific unit; for example, a receiving unit may also be described as "a unit that receives data query requests for a first data processing engine."

[0129] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A data processing method, characterized in that, include: Receive a data query request for a first data processing engine, wherein the first data processing engine includes a front-end node and a back-end node, the front-end node receives the data query request, and the back-end node stores the data; In the front-end node, based on the data query request, a target data processing engine is selected from the data processing engine candidate set. The target data processing engine processes the data query request. The data processing engine candidate set includes the first data processing engine and the second data processing engine. The data processing capability of the second data processing engine is higher than that of the first data processing engine. If the target data processing engine is the first data processing engine, then in the front-end node, the data query request is parsed and the parsing result is sent to the back-end node so that the back-end node can perform data query. If the target data processing engine is the second data processing engine, then in the front-end node, target information is generated based on the data query request and sent to the second data processing engine so that the second data processing engine can obtain data from the first data processing engine for data query. The target information is the data query request; after generating target information based on the data query request and sending it to the second data processing engine, the method further includes: determining whether the second data processing engine caches metadata, wherein the metadata is metadata of the data table storing the data queried by the data query request; if so, in the second data processing engine, using the metadata, data is pulled from the backend node, and data query is performed on the pulled data; The data in the backend node is stored according to data shards, and the metadata includes the metadata of the data shards; the step of using the metadata to pull data from the backend node includes: generating at least one data pull task according to the data shards, wherein one data pull task pulls data corresponding to at least one data shard; and executing the at least one data pull task to pull data from the backend node. In the front-end node, selecting a target data processing engine from a candidate set of data processing engines based on the data query request includes: in the front-end node, determining a predicted value of resource consumption during the processing of the data query request; and in response to determining that the predicted value of resource consumption is greater than a preset resource consumption threshold, selecting a second data processing engine as the target data processing engine.

2. The method according to claim 1, characterized in that, In the front-end node, selecting a target data processing engine from the candidate set of data processing engines based on the data query request includes: In the front-end node, the storage information corresponding to the data queried by the data query request is determined, wherein the storage information includes at least one of the following: the space of the storage table, the number of partitions, and the number of files; In response to determining at least one of the following, a second data processing engine is selected as the target data processing engine: the space of the storage table is greater than a preset space threshold, the number of partitions is greater than a preset partition number threshold, and the number of files is greater than a preset file number threshold.

3. The method according to claim 1 or 2, characterized in that, In the front-end node, generating target information based on the data query request and sending it to the second data processing engine includes: In the front-end node, the data query request is parsed to obtain a first dataset, wherein the first dataset can be processed by the first data processing engine; Determine whether the first dataset can be converted into a second dataset, wherein the second dataset can be processed by the second data processing engine; If so, the first dataset is converted into the second dataset, and the second dataset is sent as the target information to the second data processing engine.

4. The method according to claim 3, characterized in that, After determining whether the first dataset can be converted into the second dataset, the method further includes: If not, the data query request will be sent as target information to the second data processing engine.

5. The method according to claim 1, characterized in that, The data query request includes the engine identifier of the specified data processing engine; as well as In the front-end node, selecting a target data processing engine from the candidate set of data processing engines based on the data query request includes: Select the data processing engine specified in the data query request from the candidate set of data processing engines as the target data processing engine.

6. The method according to claim 5, characterized in that, In the front-end node, generating target information based on the data query request and sending it to the second data processing engine includes: In the front-end node, the data query request is sent as target information to the second data processing engine.

7. The method according to claim 1, characterized in that, The method further includes: If the second data processing engine does not cache metadata, then the metadata is obtained from the front-end node and cached in the memory of the second data processing engine. In the second data processing engine, the metadata is used to retrieve data from the first data processing engine for data querying.

8. The method according to claim 1, characterized in that, The step of parsing the data query request in the front-end node and sending the parsing result to the back-end node so that the back-end node can perform data query includes: In the front-end node, the data query request is parsed to generate a physical execution plan, which is then split into multiple plan fragments. Fragment instances are created based on the multiple plan fragments, and the fragment instances are sent to the back-end node. In the backend node, the fragment instance is processed using a pipeline link to query the data.

9. The method according to claim 1, characterized in that, The method further includes: In the front-end node, the processing results for the data query request are obtained from the second data processing engine.

10. The method according to claim 1, characterized in that, The first data processing engine includes a massively parallel processing database, and the second data processing engine includes a distributed computing framework.

11. A data processing apparatus, characterized in that, include: A receiving unit is configured to receive a data query request for a first data processing engine, wherein the first data processing engine includes a front-end node and a back-end node, the front-end node receives the data query request, and the back-end node stores data; The selection unit is used in the front-end node to select a target data processing engine from a candidate set of data processing engines based on the data query request. The target data processing engine processes the data query request. The candidate set of data processing engines includes a first data processing engine and a second data processing engine. The data processing capability of the second data processing engine is higher than that of the first data processing engine. The first processing unit is configured to, if the target data processing engine is the first data processing engine, parse the data query request in the front-end node and send the parsing result to the back-end node so that the back-end node can perform a data query. The second processing unit is configured to, if the target data processing engine is the second data processing engine, generate target information in the front-end node based on the data query request and send it to the second data processing engine, so that the second data processing engine can obtain data from the first data processing engine for data query. The target information is the data query request, and the data processing device further includes: The determining unit is used to determine whether the second data processing engine caches metadata, wherein the metadata is the metadata of the data table storing the data queried by the data query request; The first acquisition unit is used to, if the second data processing engine caches the metadata, use the metadata in the second data processing engine to pull data from the backend node and perform data query on the pulled data; Data in the backend nodes is stored according to data shards, and the metadata includes metadata of the data shards; data is pulled from the backend nodes using the metadata in the following manner: at least one data pull task is generated according to the data shards, wherein one data pull task pulls data corresponding to at least one data shard; the at least one data pull task is executed to pull data from the backend nodes; The selection unit is used to determine the predicted resource consumption value in the process of processing the data query request in the front-end node; in response to determining that the predicted resource consumption value is greater than a preset resource consumption threshold, a second data processing engine is selected as the target data processing engine.

12. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images