Task execution method and apparatus, device and storage medium

Abstract

The present application relates to the field of computer technologies, and discloses a task execution method and apparatus, a device and a storage medium. The method is executed by a first component, and comprises: receiving a data processing task, wherein the data processing task is a task to be collaboratively processed by a plurality of computing engines; splitting the data processing task on the basis of a plurality of data sources registered on the first component to obtain a plurality of data processing sub-tasks; determining execution correspondences between the plurality of data processing sub-tasks and the plurality of computing engines; and on the basis of the execution correspondences, distributing the plurality of data processing sub-tasks to task execution engines among the plurality of computing engines.

Description

Task execution method, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 2, 2023, with application number 202310966040.0 and invention name “Task execution method, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of computer technology, and in particular to a task execution method, apparatus, device, and storage medium.

[0003] Background of the Invention

[0004] Big data refers to data sets that cannot be captured, managed, and processed within a specific timeframe using conventional software tools. These data assets require new processing models to enhance decision-making, insight discovery, and process optimization. Different computing engines are used to process data based on the specific needs of big data processing. Complex data processing requires adding a variety of computing engines to the data architecture used for big data processing, enabling unified data processing across multiple engines.

[0005] In related technologies, in order to combine multiple computing engines for unified data analysis, the cross-domain computing of the computing engine is usually optimized by adjusting the interface within the computing engine so that domain names or ports other than the application domain name or port can be accessed through the computing engine.

[0006] As the volume of big data increases, considering the efficient processing characteristics of cloud computing, data is gradually processed in multi-cloud scenarios (multiple cloud environments), including hybrid cloud (hybrid cloud environment) and cross-cloud. The optimization of the above computing engines all focuses on a single computing engine, and does not fully consider the joint situation between computing engines distributed under multiple cloud platforms, and cannot effectively implement the joint analysis process across computing engines.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a task execution method, apparatus, device, and storage medium that can simplify the execution of data processing tasks by splitting them, fully utilizing different computing engines to perform more targeted execution of data processing subtasks, and thus improve task execution efficiency. The technical solution is as follows.

[0009] In one aspect, a task execution method is provided, which is executed by a first component, and the method includes:

[0010] receiving a data processing task, where the data processing task is a task to be collaboratively processed by multiple computing engines, and the first component is registered with the multiple computing engines and at least one data source respectively called by the multiple computing engines;

[0011] Splitting the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks, and the computing engine is used to call the data source to execute at least one data processing subtask;

[0012] Determining an execution correspondence between the plurality of data processing subtasks and the plurality of computing engines;

[0013] Based on the execution correspondence, the multiple data processing subtasks are distributed to a task execution engine among the multiple computing engines, where the task execution engine is a computing engine for executing the data processing subtasks.

[0014] In another aspect, a data processing device is provided, comprising:

[0015] a task receiving module, configured to receive a data processing task, wherein the data processing task is a task to be collaboratively processed by multiple computing engines, wherein the first component is registered with the multiple computing engines and at least one data source respectively called by the multiple computing engines;

[0016] a task splitting module, configured to split the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks, and the computing engine is configured to call the data source to execute at least one data processing subtask;

[0017] a relationship determination module, configured to determine an execution correspondence between the plurality of data processing subtasks and the plurality of computing engines;

[0018] A task distribution module is used to distribute the multiple data processing subtasks to a task execution engine among the multiple computing engines based on the execution correspondence, and the task execution engine is a computing engine used to execute the data processing subtasks.

[0019] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a task execution method as described in any of the above-mentioned embodiments of the present application.

[0020] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a task execution method as described in any of the above-mentioned embodiments of the present application.

[0021] On the other hand, a computer program product or computer program is provided, which includes computer instructions, and the computer instructions are loaded and executed by a processor to implement the task execution method as described in any of the above embodiments of the present application.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0025] FIG2 is a flowchart of a task execution method provided by an exemplary embodiment of the present application;

[0026] FIG3 is a flowchart of a task execution method provided by another exemplary embodiment of the present application;

[0027] FIG4 is a flowchart of a task execution method provided by another exemplary embodiment of the present application;

[0028] FIG5 is a schematic diagram of an architecture for distributing multiple data processing subtasks to a task execution engine according to an exemplary embodiment of the present application;

[0029] FIG6 is a flowchart of a task execution method provided by another exemplary embodiment of the present application;

[0030] FIG7 is a schematic diagram of an architecture for registering a computing engine and a data source to a first component provided by an exemplary embodiment of the present application;

[0031] FIG8 is a flowchart of sending task execution results provided by an exemplary embodiment of the present application;

[0032] FIG9 is a flow chart of a data routing algorithm provided by an exemplary embodiment of the present application;

[0033] FIG10 is a structural block diagram of a task execution device provided by an exemplary embodiment of the present application;

[0034] FIG11 is a structural block diagram of a task execution device provided by another exemplary embodiment of the present application;

[0035] FIG12 is a structural block diagram of a server device provided by an exemplary embodiment of the present application.

[0036] Implementation Method

[0037] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0038] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0039] Hybrid cloud: A form of cloud computing that combines private and public clouds to improve cross-cloud resource utilization. Hybrid cloud helps users manage information technology (IT) infrastructure across clouds and geographies. It's an integrated system that encompasses resources and products from both public and private clouds.

[0040] Multi-cloud: Using multiple cloud computing services from multiple cloud providers (both private and public) in a heterogeneous environment. A multi-cloud strategy provides greater flexibility and reduces risk. Choose the services best suited for a specific task from different cloud providers, or leverage services from a specific cloud provider in a specific location.

[0041] Structured Query Language (SQL): SQL is a computer language used to store, retrieve, and modify data stored in relational data sources. Data sources are systems used to store and process data, such as traditional relational databases (MySQL), object-relational databases (PostgreSQL), click stream data sources (ClickHouse), or data warehouse tools (Hive) for big data NoSQL systems, distributed data sources (Hadoop Database, HBase), and distributed search engines (Elasticsearch).

[0042] In related technologies, in order to combine multiple computing engines for unified data analysis, the cross-domain computing of the computing engine is usually optimized by adjusting the interface within the computing engine so that domain names or ports other than the application domain name or port can be accessed through the computing engine. With the increase in the volume of big data, considering the efficient processing characteristics of cloud computing, data is gradually processed in multi-cloud (multiple cloud environments) scenarios, including hybrid cloud (hybrid cloud environment), cross-cloud, etc. The optimization of the above computing engines is focused on a single computing engine, and does not fully consider the joint situation between computing engines distributed under multiple cloud platforms, and cannot effectively implement the joint analysis process across computing engines.

[0043] In an embodiment of the present application, a data processing task to be collaboratively processed by multiple computing engines is received, metadata corresponding to the multiple computing engines are obtained, and then the data processing task is split based on the metadata to obtain multiple data processing subtasks; based on the execution correspondence between the multiple data processing subtasks and the multiple computing engines, the multiple data processing subtasks are distributed to the task execution engine to execute the data processing subtasks. The data processing task is split with the help of multiple data sources to determine the data processing subtasks executed by different computing engines according to the calling situation of the computing engine on the data source. By splitting the data processing task, the way of executing the data processing task is simplified, and different computing engines are fully utilized to perform more targeted execution processing on the data processing subtasks, thereby improving the efficiency of task execution. The above-mentioned task execution method can be applied to various data processing scenarios such as sales data processing scenarios, medical data processing scenarios, financial data processing scenarios, and traffic data processing scenarios. The above scenarios are only illustrative examples, and the embodiments of the present application are not limited to this.

[0044] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the data processing tasks, data sources, and other content involved in this application were obtained with full authorization.

[0045] Next, the implementation environment involved in the embodiments of this application is described. The task execution method provided in the embodiments of this application can be implemented by a terminal alone, by a server device, or by a terminal and a server device through data interaction, and this embodiment of the application is not limited thereto. The task execution method interactively executed by a terminal and a server device is used as an example for description.

[0046] Schematically, please refer to FIG. 1 , the implementation environment involves a terminal 110 and a first component 120 , and the terminal 110 and the first component 120 are connected via a communication network 130 .

[0047] In some embodiments, the terminal 110 corresponds to multiple functions to implement functions such as searching for files, chatting, playing audio, etc. In the process of implementing the above functions, it is necessary to process data related to the functions.

[0048] Terminal 110 receives a function triggering operation and generates a data processing task based on the function triggering operation. A data processing task represents a task for processing data related to the function. A data processing task is a task that needs to be completed by a management component. Terminal 110 sends the data processing task to first component 120 via communication network 130, so that first component 120 can implement the data processing process.

[0049] Illustratively, the first component 120 receives a data processing task, wherein the data processing task is a task to be collaboratively processed by multiple computing engines.

[0050] Among them, multiple computing engines and multiple data sources called by the computing engines are registered on the first component.

[0051] In some embodiments, the first component 120 splits the data processing task based on multiple data sources registered on the first component to obtain multiple data processing subtasks.

[0052] The first component 120 splits the data processing task based on the attribute information represented by the multiple data sources, so as to determine the calling status of the data source when processing the data processing subtasks.

[0053] The computing engine is used to call the data source to execute at least one data processing subtask.

[0054] Illustratively, when executing a data processing subtask via a data source, the data processing subtask is performed by invoking the data source's computing engine. For example, if data processing subtask 1 is executed via data source 1, and data source 1 is implemented via computing engine A1 when executing data processing subtask 1, then computing engine A1 executes data processing subtask 1 by invoking data source 1.

[0055] In some embodiments, the first component 120 determines the execution correspondence between the plurality of data processing subtasks and the plurality of computing engines. Schematically, the first component 120 analyzes the execution of each computing engine calling a data source to execute the data processing subtask, thereby determining the execution correspondence between the plurality of data processing subtasks and the plurality of computing engines.

[0056] In some embodiments, multiple data processing subtasks are distributed to corresponding task execution engines based on execution correspondences, where a task execution engine is a computing engine used to execute the data processing subtasks. For example, if the execution correspondence indicates that computing engine A1 executes data processing subtask 1 by calling data source 1, then computing engine A1 is responsible for executing data processing subtask 1. Based on this execution correspondence, data processing subtask 1 is distributed to computing engine A1, and so on.

[0057] The multiple computing engines each receive at least one data processing subtask, and each process the data on the received at least one data processing subtask, thereby obtaining data processing results corresponding to the multiple data processing subtasks. Illustratively, first component 120 transmits the multiple data processing results to terminal 110 via communication network 130; alternatively, first component 120 aggregates the multiple data processing results to obtain a task execution method, and transmits the task execution method to terminal 110 via communication network 130.

[0058] It is worth noting that the above-mentioned terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptop computers, intelligent voice interaction devices, smart home appliances, and vehicle-mounted terminals, and can also be implemented as desktop computers, etc. The first component is configured in multiple cloud environments, such as: a cloud environment is an environment developed under a cloud platform, which includes at least one cloud server device, and a multi-cloud environment (multiple cloud environments) is a communication environment in which multiple cloud platforms communicate with each other, and a multi-cloud environment includes multiple cloud server devices. For example: the first component is a component configured in any cloud server device in a multi-cloud environment; or the first component is a separate component configured in a multi-cloud environment, etc.

[0059] Cloud server devices are used to provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0060] Cloud technology refers to a hosting technology that unifies hardware, applications, networks, and other resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology encompasses network, information, integration, management platform, and application technologies, all based on the cloud computing business model. It can be used to create a flexible and convenient resource pool for on-demand use.

[0061] In some embodiments, the above-mentioned server device can also be implemented as a node in a blockchain system.

[0062] In conjunction with the above-mentioned noun introduction and application scenarios, the task execution method provided by this application is described. The method provided by the embodiment of this application can be applied to various scenarios such as cloud technology, artificial intelligence, and smart transportation. Taking the application of this method to the first component as an example, as shown in Figure 2, the method includes the following steps 210 to 240.

[0063] Step 210: Receive a data processing task.

[0064] Schematically, the data processing task is used to perform targeted data processing processes to achieve data processing effects.

[0065] The data processing task is used to implement function triggering, for example: the data processing task is used to determine the interface content displayed after triggering control A on the terminal; or, the data processing task is used to implement data acquisition, for example: the data processing task is used to obtain relevant data information through instruction 1, etc.

[0066] The data processing task is a task sent by a client; or, the data processing task is a task sent by a server device, etc.

[0067] Among them, data processing tasks are tasks to be processed collaboratively by multiple computing engines.

[0068] Schematically, a computing engine is a program used to process data. Data is stored in a data source (also called a database). Traditionally, the storage engine used for data storage and the computing engine used for data computation are integrated within the data source. However, with the advent of the big data era and the increasing volume of data required for computation, the storage engine and computing engine have evolved separately to increase the flexibility of data source processing. Therefore, the term "computing engine" is used to represent the program that processes data in the data source.

[0069] Each computing engine is used to call at least one data source. Schematically, computing engine 1 calls data sources A and B, computing engine 2 calls data source C, computing engine 3 calls data source A and D, and so on.

[0070] In some embodiments, the data processing task is received by the first component, and the data processing task is a task that needs to be collaboratively processed by multiple computing engines.

[0071] Among them, multiple computing engines and multiple data sources called by the computing engines are registered on the first component.

[0072] Based on the registration process, the first component can roughly know the engine information of multiple computing engines and the data source information corresponding to multiple data sources, so that the first component can determine how to call the data source through the computing engine under the data processing task to achieve the purpose of executing the data processing task.

[0073] Step 220: Split the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks.

[0074] Illustratively, after multiple data sources are registered on the first component, the first component can determine the attribute information corresponding to the multiple data sources respectively based on the registration process, and then split the data processing tasks based on the attribute information corresponding to the multiple data sources respectively.

[0075] The computing engine is used to call the data source to execute at least one data processing subtask.

[0076] Illustratively, when executing a data processing subtask through a data source, the data processing process is performed by a computing engine used to process the data in the data source. For example, multiple data sources include data source 1, data source 2, and data source 3. Data source 1 performs data processing through the call of computing engine M, while data sources 2 and 3 perform data processing through the call of computing engine N, and so on.

[0077] In some embodiments, the splitting process of data processing subtasks is not only related to the data source, but also to the computing engine that calls the data source. Schematically, the data processing task is split based on the call correspondence between multiple data sources when executing the data processing task.

[0078] The call correspondence relationship is used to represent the relationship between the data source and the computing engine.

[0079] In some embodiments, based on multiple computing engines registered on the first component, a call correspondence relationship between multiple data sources and multiple computing engines is determined.

[0080] Step 230: Determine the execution correspondence between the plurality of data processing subtasks and the plurality of computing engines.

[0081] Schematically, based on the computing engine used to call the data source to execute at least one data processing subtask, it is possible to determine the computing engine used to process the data subtask, and then determine the computing engine used to process multiple data processing subtasks separately, that is, determine the execution correspondence between multiple data processing subtasks and multiple computing engines.

[0082] For example, computing engine M executes data processing subtask A1 by calling data source 1, which means that there is an execution correspondence between computing engine M and data processing subtask A1; computing engine N executes data processing subtask A2 by calling data source 2 and data source 3, which means that there is an execution correspondence between computing engine N and data processing subtask A2, and so on.

[0083] In some embodiments, a computing engine is capable of performing at least one data processing subtask.

[0084] Illustratively, the computing engine M is used to execute the data processing subtask A1; or, the computing engine M is used to execute the data processing subtask A1 and the data processing subtask A3.

[0085] For example: computing engine M executes data processing subtask A1 by calling data source 1; or, computing engine M executes data processing subtask A1 by calling data source 1. In addition, computing engine M executes data processing subtask A3 by calling data source 1 and data source 4, etc.

[0086] Step 240 : Distribute the multiple data processing subtasks to the task execution engines in the multiple computing engines based on the execution correspondence.

[0087] Schematically, after determining the execution correspondence between multiple data processing subtasks and multiple computing engines, each data processing subtask is analyzed separately to determine the computing engine that has an execution correspondence with each data processing subtask, and then the data processing subtask is sent to the computing engine that has an execution correspondence with it.

[0088] Among them, the task execution engine is a computing engine used to execute data processing subtasks.

[0089] The computing engine that has an execution correspondence relationship with the data processing subtask is called a task execution engine. The above-mentioned determination of the computing engine that has an execution correspondence relationship with each data processing subtask is to determine the task execution engine corresponding to each data processing subtask, so as to send (distribute) multiple data processing subtasks to the corresponding task execution engines respectively, thereby executing at least one received data processing subtask through the task execution engine.

[0090] Schematically, multiple data processing subtasks include data processing subtask A1 and data processing subtask A2. Based on the execution correspondence, the task execution engine corresponding to data processing subtask A1 is determined to be computing engine M, and the task execution engine corresponding to data processing subtask A2 is determined to be computing engine N; then, when distributing multiple data processing subtasks to the task execution engine used to execute the data processing subtasks, data processing subtask A1 is sent to computing engine M, and data processing subtask A2 is sent to computing engine N.

[0091] In summary, the first component receives a data processing task to be collaboratively processed by multiple computing engines, and splits the data processing task based on multiple data sources registered on the first component, wherein the computing engine is used to call the data source to execute at least one data processing subtask, and then distributes the multiple data processing subtasks to the multiple computing engines based on the execution correspondence between the multiple data processing subtasks and the multiple computing engines. By splitting the data processing task with the help of multiple data sources, the data processing subtasks executed by different computing engines are determined according to the calling situation of the computing engine on the data source. By splitting the data processing task, the method of executing the data processing task is simplified, and different computing engines are fully utilized to perform more targeted execution processing on the data processing subtasks, thereby improving the efficiency of task execution.

[0092] In one embodiment, data processing tasks are split using the call correspondence between the data source and the computing engine, and the data processing subtasks are distributed using the execution correspondence between the computing engine and the data processing subtasks. As shown in FIG3 , the embodiment shown in FIG2 can also be implemented as steps 310 to 350; step 220 shown in FIG2 can also be implemented as steps 320 to 330.

[0093] Step 310: Receive a data processing task.

[0094] The data processing task is a task to be collaboratively processed by multiple computing engines, and multiple computing engines and multiple data sources called by the computing engines are registered on the first component.

[0095] In some embodiments, the first component is a component deployed in a first cloud environment. A large number of computing engines are deployed in the first cloud environment. The large number of computing engines deployed in the first cloud environment are registered on the first component. The first component can coordinate and manage the large number of computing engines in the first cloud environment.

[0096] Multiple computing engines for data processing tasks to be collaboratively processed by multiple computing engines are included in a large number of computing engines within the first cloud environment; after receiving the data processing task, the first component collaboratively processes the data processing task by managing multiple computing engines in the large number of computing engines for executing the data processing task.

[0097] The plurality of computing engines for collaboratively processing data processing tasks by the plurality of computing engines are partially included in a large number of computing engines within a first cloud environment, and partially included in a large number of computing engines outside the first cloud environment, for example, partially included in a large number of computing engines within a second cloud environment. Upon receiving a data processing task, the first component manages a portion of the computing engines within the first cloud environment and another portion of the computing engines within the second cloud environment based on the registration status of the plurality of computing engines with the first component, thereby enabling collaborative processing of the data processing task by the plurality of computing engines.

[0098] In some embodiments, based on the fact that multiple computing engines are registered on the first component and multiple data sources are called by the computing engines, the registration relationship between the first component and the multiple computing engines is called a first registration relationship, and the registration relationship between the first component and the multiple data sources is called a second registration relationship.

[0099] A first registration relationship exists between a first component and multiple computing engines used to collaboratively process data processing tasks. The first registration relationship is used to characterize the data sources called by the computing engines. For example, if computing engines 1 and 2 are registered with the first component, the first registration relationship between the first component and computing engine 1 indicates that computing engine 1 is used to call data sources A and B; the first registration relationship between the first component and computing engine 2 indicates that computing engine 2 is used to call data source C, and so on.

[0100] A second registration relationship exists between the first component and multiple data sources invoked by the computing engine. This second registration relationship is used to represent data source information corresponding to the data source (e.g., metadata corresponding to the data source). For example, if data source A and data source B are registered with the first component, the second registration relationship between the first component and data source A indicates the properties of data source A; the second registration relationship between the first component and data source B indicates the properties of data source B, etc.

[0101] Step 320: Determine the call correspondence between the multiple data sources and the multiple computing engines based on the multiple computing engines registered on the first component.

[0102] The call correspondence relationship is used to represent the relationship between the data source and the computing engine.

[0103] Based on the registration of multiple computing engines on the first component, the first component obtains engine information sent by the multiple computing engines, where the engine information is used to represent the data source called by the computing engine.

[0104] Schematically, the first component reads engine information corresponding to the computing engines through interfaces respectively configured for the multiple computing engines.

[0105] The first component obtains engine information corresponding to the multiple computing engines respectively, and thereby determines the calling correspondence between the multiple data sources and the multiple computing engines based on the situation in which the data source represented by the engine information is called by the computing engine.

[0106] Schematically, if data source 1 is called by computing engine M, it means that there is a calling correspondence between data source 1 and computing engine M; if data source 2 is called by computing engine N, it means that there is a calling correspondence between data source 2 and computing engine N; if data source 3 is also called by computing engine N, it means that there is a calling correspondence between data source 3 and computing engine N, and so on.

[0107] In one embodiment, metadata corresponding to multiple data sources are obtained.

[0108] Illustratively, after multiple data sources are registered on the first component, the first component can obtain metadata corresponding to the multiple data sources respectively, where the metadata is used to represent attribute information corresponding to the data sources.

[0109] Based on the data source used to store data, metadata is used to describe the data source. The attribute information represented by metadata includes data source type information (such as: the type of data stored by the data source), data source volume information (such as: the amount of data stored by the data source), data source storage location (such as: the storage location of the data source), data source historical access information, data source log information, and at least one of the multiple information used to describe the data source.

[0110] In some embodiments, after multiple data sources are registered on the first component, the first component obtains metadata corresponding to the multiple data sources through interfaces on the multiple data sources that communicate with the first component.

[0111] Schematically, after data source 1 is registered with the first component, the first component can obtain metadata corresponding to data source 1 from interface a on data source 1, where interface a is an interface for obtaining metadata pre-agreed with the first component based on the registration process.

[0112] Multiple data sources periodically determine the metadata they represent. After data source 1 is registered with the first component, the first component can obtain the metadata corresponding to data source 1 in the most recent period from interface a on data source 1; or, after data source 1 is registered with the first component, the first component sends a metadata acquisition request to data source 1 through interface a on data source 1, thereby triggering data source 1 to determine the corresponding metadata and send the metadata corresponding to data source 1 to the first component, etc.

[0113] Step 330: Based on the correspondence between the multiple data sources registered on the first component and the call, the data processing task is split to obtain multiple data processing subtasks.

[0114] After determining the multiple data sources registered on the first component and the call correspondence between the multiple data sources and the multiple computing engines, the data processing tasks are split based on the multiple data sources and the call correspondence.

[0115] Based on the correspondence between the multiple data sources registered on the first component and the calls, a task processing part implemented by at least one data source under the same computing engine when executing the data processing task is determined.

[0116] Schematically, computing engine M can implement task processing part a in data processing task A by calling data source 1, and computing engine N can implement task processing part b in data processing task A by calling data source 2; or, computing engine M can implement task processing part a in data processing task A by calling data source 1, and computing engine N can implement task processing part b in data processing task A by calling data source 2 and data source 3, etc.

[0117] The data processing task is split based on the task processing parts corresponding to multiple computing engines to obtain multiple data processing subtasks.

[0118] Schematically, based on the fact that computing engine M is used to complete task processing part a and computing engine N is used to complete task processing part b, the data processing task can be split based on this division process to obtain data processing subtask A1 representing task processing part a and data processing subtask A2 representing task processing part b.

[0119] In one embodiment, metadata corresponding to multiple data sources are obtained.

[0120] Illustratively, after multiple data sources are registered on the first component, the first component can obtain metadata corresponding to the multiple data sources respectively, where the metadata is used to represent attribute information corresponding to the data sources.

[0121] In some embodiments, after multiple data sources are registered on the first component, the first component obtains metadata corresponding to the multiple data sources through interfaces on the multiple data sources that communicate with the first component.

[0122] Schematically, after data source 1 is registered with the first component, the first component can obtain metadata corresponding to data source 1 from interface a on data source 1, where interface a is an interface for obtaining metadata pre-agreed with the first component based on the registration process.

[0123] Multiple data sources periodically determine the metadata they represent. After data source 1 is registered with the first component, the first component can obtain the metadata corresponding to data source 1 in the most recent period from interface a on data source 1; or, after data source 1 is registered with the first component, the first component sends a metadata acquisition request to data source 1 through interface a on data source 1, thereby triggering data source 1 to determine the corresponding metadata and send the metadata corresponding to data source 1 to the first component, etc.

[0124] In one embodiment, the data processing task is split based on metadata to obtain multiple data processing subtasks.

[0125] The computing engine is used to call the data source to execute at least one data processing subtask.

[0126] Illustratively, after obtaining metadata corresponding to multiple data sources, multiple metadata are obtained, and the multiple metadata correspond to the multiple data sources one by one; the multiple metadata are analyzed respectively to split the data processing tasks.

[0127] In one embodiment, the data processing task is sent to at least one data source among the multiple data sources.

[0128] Wherein, at least one data source is used to filter the data stored in the data source through a data processing task to obtain task data.

[0129] For first metadata among the multiple metadata, a data source corresponding to the first metadata is determined.

[0130] Illustratively, the first metadata is any one of the plurality of metadata, that is, any one of the metadata is used as the first metadata for analysis. Based on the attribute information represented by the first metadata, a data source corresponding to the first metadata is determined from the plurality of data sources.

[0131] Send the data processing task to the data source corresponding to the first metadata.

[0132] The data source corresponding to the first metadata is used to filter the data stored in the data source through the data processing task to obtain task data.

[0133] Illustratively, when splitting data processing tasks, an operator pushdown method is used. By dispatching the data processing tasks to multiple data sources, the data sources' data processing capabilities are leveraged to perform preliminary data screening. For example, if the data processing task is "Get the names of objects over 5 years old and taller than 1 meter," the data processing task is dispatched to each data source. For example, if analysis is performed on any one of the multiple data sources, and that data source stores a large amount of age information, the first data source can perform preliminary data screening based on the data processing task, such as deleting objects under 5 years old, thereby filtering the data stored in that data source.

[0134] Based on the above-mentioned data filtering process, the tediousness of the computing engine calling invalid data stored under the data source can be avoided. For example: if the above-mentioned data filtering process is not adopted, when data source 1 is called by computing engine M, data source 1 needs to send the large amount of data stored therein to computing engine M so that computing engine M can perform data screening; if the above-mentioned data filtering process is adopted, data source 1 will first perform a preliminary screening of the data based on the data processing task, and when called by computing engine M, data source 1 will send the screened data to computing engine M so that computing engine M can perform faster data analysis on the screened data. The above-mentioned process can not only avoid the problem of invalid data calling, but also reduce the traffic of network data transmission and avoid the problem of large data transmission volume.

[0135] In one embodiment, the data processing task is split based on the task data to obtain multiple data processing subtasks.

[0136] Illustratively, task data corresponding to the plurality of data sources are determined based on the above process, where the task data represents data obtained after filtering the data sources.

[0137] Based on the task data and the call correspondence between multiple data sources and multiple computing engines, the data processing task is split to obtain multiple data processing subtasks.

[0138] Step 340: Determine the execution correspondence between the multiple data processing subtasks and the multiple computing engines.

[0139] Schematically, based on the computing engine used to call the data source to execute at least one data processing subtask, it is possible to determine the computing engine used to process the data subtask, and then determine the computing engine used to process multiple data processing subtasks separately, that is, determine the execution correspondence between multiple data processing subtasks and multiple computing engines.

[0140] In some embodiments, a computing engine is capable of performing at least one data processing subtask.

[0141] Step 350 : Distribute the multiple data processing subtasks to the task execution engines in the multiple computing engines based on the execution correspondence.

[0142] Schematically, after determining the execution correspondence between multiple data processing subtasks and multiple computing engines, each data processing subtask is analyzed separately to determine the computing engine that has an execution correspondence with each data processing subtask, and then the data processing subtask is sent to the computing engine that has an execution correspondence with it.

[0143] The task execution engine is a computing engine used to execute data processing subtasks, that is, a computing engine that has an execution correspondence relationship with a data processing subtask is called a task execution engine.

[0144] The plurality of data processing subtasks are respectively sent (distributed) to a task execution engine for executing the data processing subtasks, so that at least one received data processing subtask is executed by the task execution engine.

[0145] To sum up, data processing tasks are split with the help of multiple data sources to determine the data processing subtasks executed by different computing engines based on the computing engine's call to the data source. By splitting the data processing tasks, the way of executing data processing tasks is simplified, and different computing engines are fully utilized to perform more targeted execution of data processing subtasks, thereby improving task execution efficiency.

[0146] In the embodiment of the present application, the content of splitting the data processing task by obtaining the metadata corresponding to the data source is introduced. Based on the registration of multiple data sources in the first component, the first component can obtain metadata used to characterize the attribute information of the data source, and then use the metadata to split the data processing task in a more targeted manner to determine the task parts processed by different computing engines by calling the data source, thereby allocating data processing subtasks to different computing engines; with the help of metadata, the deployment status of multiple data sources can be expressed more intuitively and accurately, and the received data processing subtasks can be analyzed separately by multiple computing engines to improve the processing efficiency of data processing tasks.

[0147] In one embodiment, a first component for executing the task execution method is deployed within a first cloud environment. After determining a corresponding task execution engine for any data processing subtask, a method for sending the data processing subtask to the task execution engine is determined based on the deployment location of the task execution engine. As shown schematically in FIG4 , step 240 shown in FIG2 can also be implemented as steps 410 to 442.

[0148] Step 410: Determine the task execution engines corresponding to the plurality of data processing subtasks based on the execution correspondence.

[0149] Illustratively, after determining the execution correspondences between multiple data processing subtasks and multiple computing engines, each data processing subtask is analyzed separately to determine the computing engine with which each data processing subtask has an execution correspondence. This computing engine with such an execution correspondence is then designated as the task execution engine corresponding to the data processing subtask. In other words, a task execution engine is a computing engine used to execute a data processing subtask.

[0150] In some embodiments, multiple data processing subtasks are distributed to corresponding task execution engines based on the deployment relationship between the deployment location of the task execution engine and the first cloud environment.

[0151] A first component for performing data processing is deployed within the first cloud environment. The term "cloud" is a metaphor for the Internet or the Internet of Things (IoT). A cloud environment refers to an internet or big data environment that can provide computing power, storage capacity, or virtual machine services on demand to users or various application systems from a dynamic virtualized resource pool.

[0152] Schematically, various data functions are implemented in a cloud environment. Within the cloud environment, various functional service architectures for data processing are deployed. Computing engines and data sources are deployed within the cloud environment. Schematically, the cloud environment is implemented in various cloud computing formats, such as private clouds and public clouds.

[0153] In some embodiments, multiple computing engines for performing data processing tasks are deployed in at least one cloud environment.

[0154] Schematically, when multiple computing engines for executing data processing tasks are deployed in multiple cloud environments, the data processing task executes data processing in multiple cloud environments. Because a task execution engine represents the computing engine corresponding to the data processing subtask, the deployment location of the task execution engine represents the deployment location of the computing engine.

[0155] At least one cloud environment includes the above-mentioned first cloud environment, and the large number of computing engines deployed in the first cloud environment include all or part of the computing engines used to perform data processing tasks.

[0156] If at least one cloud environment does not include the above-mentioned first cloud environment, then the large number of computing engines deployed in the first cloud environment does not include a computing engine for performing data processing tasks, and multiple computing engines for performing data processing tasks establish a relationship with the first component through registration.

[0157] The deployment location is content predetermined according to the deployment status of the computing engine; and the deployment relationship is used to characterize the inclusion status between the deployment location and the first cloud environment.

[0158] Illustratively, the deployment location indicates that the task execution engine (computing engine) is deployed in the first cloud environment; or, the deployment location indicates that the task execution engine (computing engine) is deployed in another cloud environment outside the first cloud environment.

[0159] In one embodiment, the data processing subtask is sent to the corresponding task execution engine using a corresponding sending method according to the deployment relationship between the deployment location of the task execution engine and the first cloud environment.

[0160] Schematically, as shown in the following steps 421 to 422, 431 to 432, and 441 to 442, a processing flow is provided for sending data processing subtasks to corresponding task execution engines according to three deployment relationships.

[0161] Step 421 : In response to the first task execution engine being deployed in the first cloud environment, at least one data processing subtask executed by the first task execution engine is determined from a plurality of data processing subtasks as a first data processing subtask.

[0162] Illustratively, the first task execution engine is any computing engine to be analyzed. For example, a task execution engine for executing a data processing subtask is determined based on the data processing subtask. Taking the task execution engine as the first task execution engine as an example, the deployment location of the first task execution engine is determined, that is, the cloud environment in which the first task execution engine is located within the at least one cloud environment is determined.

[0163] The at least one cloud environment includes a first cloud environment, and the first task execution engine is deployed in the first cloud environment, that is, the first task execution engine is deployed in the first cloud environment. At least one data processing subtask executed by the first task execution engine is determined from the plurality of data processing subtasks as the first data processing subtask.

[0164] The first data processing subtask is used to represent a data processing subtask processed by a computing engine deployed in the first cloud environment.

[0165] Schematically, the first task execution engine is a task execution engine determined when executing data processing subtask A1, and data processing subtask A1 is used as the above-mentioned first data processing subtask; in addition, other data processing subtasks executed by the first task execution engine can be determined based on the first task execution engine and the execution correspondence relationship. The other data processing subtasks are also data processing subtasks executed by the first task execution engine by calling the data source in the first cloud environment, and the other data processing subtasks are used as the first data processing subtask.

[0166] That is, the first data processing subtask is implemented as at least one data processing subtask processed by a computing engine deployed in the first cloud environment.

[0167] Step 422: Send the first data processing subtask to the first task execution engine.

[0168] Illustratively, after determining the first task execution engine and the corresponding first data processing subtask, the first component is deployed in the first cloud environment, so the first component can send the first data processing subtask to the first task execution engine.

[0169] As shown in FIG5 , it is a schematic diagram of an architecture for distributing multiple data processing subtasks to a task execution engine, which includes three cloud environments, namely, cloud environment 510 , cloud environment 520 , and cloud environment 530 .

[0170] Taking cloud environment 520 as the first cloud environment as an example, a first component 521 is deployed in the cloud environment 520, and multiple data sources are registered in the first component (such as the Hive data source and iceberg data source shown in the figure); after the first component 521 is split to obtain multiple data processing subtasks, for the first task execution engine 522 corresponding to any data processing subtask (such as: the first task execution engine is implemented as Spark or Presto, etc.), when the first task execution engine 522 is deployed in the first cloud environment, at least one data processing subtask executed by the first task execution engine 522 is determined from the multiple data processing subtasks as the first data processing subtask, and then the first data processing subtask is distributed to the first task execution engine 522.

[0171] The cloud environment 510 is implemented as a distributed file system (Hadoop Distributed System, HDFS); the cloud environment 520 is implemented as an object storage (Cloud Object Storage, COS); the cloud environment 530 is implemented as an object storage service (Object Storage Service, OSS), etc. The above are only illustrative examples and the embodiments of the present application are not limited to this.

[0172] Step 431 : In response to the second task execution engine being deployed in the second cloud environment, at least one data processing subtask executed by the second task execution engine is determined from a plurality of data processing subtasks as a second data processing subtask.

[0173] Illustratively, the second task execution engine is any computing engine to be analyzed. For example, a task execution engine for executing a data processing subtask is determined based on the data processing subtask. Taking the task execution engine as the second task execution engine as an example, the deployment location of the second task execution engine is determined, that is, the cloud environment in which the second task execution engine is located within the at least one cloud environment is determined.

[0174] The plurality of computing engines are run in at least one cloud environment, and the at least one cloud environment includes a second cloud environment that is communicatively connected to the first cloud environment.

[0175] That is, the second cloud environment is a different cloud environment from the first cloud environment. For example, the first cloud environment is a public cloud T, and the second cloud environment is a public cloud A that is communicatively connected to the public cloud T. Alternatively, the first cloud environment is a public cloud T, and the second cloud environment is a private cloud H that is communicatively connected to the public cloud T.

[0176] A second component is deployed in the second cloud environment. The second component has the same function as the first component and is capable of coordinating and managing a large number of computing engines in the cloud environment. When the second component is used to perform data processing tasks, the second component can be referred to as the first component, and the second cloud environment in which the second component is deployed can be referred to as the first cloud environment. This is merely an illustrative example and is not limited in the embodiments of the present application.

[0177] In some embodiments, when it is determined that the second task execution engine is deployed in the second cloud environment, at least one data processing subtask executed by the second task execution engine is determined from the plurality of data processing subtasks as the second data processing subtask.

[0178] The second data processing subtask is used to represent a data processing subtask processed by a computing engine deployed in the second cloud environment.

[0179] Schematically, the second task execution engine is a task execution engine determined when executing data processing subtask A2, and the data processing subtask A2 is used as the above-mentioned second data processing subtask; in addition, other data processing subtasks executed by the second task execution engine can be determined based on the second task execution engine and the execution correspondence. The other data processing subtasks are also data processing subtasks executed by the second task execution engine by calling the data source in the second cloud environment, and the other data processing subtasks are used as the second data processing subtasks.

[0180] That is, the second data processing subtask is implemented as at least one data processing subtask processed by a computing engine deployed in the second cloud environment.

[0181] Step 432: Send the second data processing subtask to the second component.

[0182] The second component is used to send the second data processing subtask to the second task execution engine.

[0183] The second cloud environment and the first cloud environment establish a communication relationship through mutual registration between the second component and the first component. Schematically, the second cloud environment registers the second component on the first component in the first cloud environment; and the first cloud environment registers the first component on the second component in the second cloud environment.

[0184] The purpose of registering the second component on the first component is to register the computing engine in the second cloud environment on the second component. By registering the second component on the first component, the first component is aware of the computing engine registered on the second component, that is, the first component is aware of the distribution of the computing engines in the second cloud environment with the help of the second component.

[0185] That is, based on the deployment of the second component for managing the computing engine in the second cloud environment, the registration engine in the second cloud environment including the second task execution engine is registered on the first component through indirect registration.

[0186] Illustratively, after determining the second task execution engine and the corresponding second data processing subtask, since the second task execution engine is deployed in the second cloud environment, the second task execution engine is registered on the second component, and the first component sends the second data processing subtask to the second component, so as to send the second data processing subtask to the second task execution engine for processing the second data processing subtask through the second component.

[0187] As shown in Figure 5, taking cloud environment 530 as the second cloud environment as an example, a second component 531 is deployed in the cloud environment 530, and multiple data sources (such as the Hive data source and Hudi data source shown in the figure) are registered in the second component; after the first component 521 splits to obtain multiple data processing subtasks, for the second task execution engine 532 corresponding to any data processing subtask (such as: the second task execution engine is implemented as Presto or StarRocks, etc.), when the second task execution engine 532 is deployed in the second cloud environment, at least one data processing subtask executed by the second task execution engine 532 is determined from the multiple data processing subtasks as the second data processing subtask; the first component 521 in the cloud environment 520 sends the split second data processing subtask to the second component 531, so as to use the second component 531's overall management function of the computing engine under the cloud environment 530 to send the second data processing subtask to the corresponding second task execution engine 532.

[0188] In some embodiments, multiple data processing subtasks are distributed to corresponding task execution engines based on the above process.

[0189] Step 441 : In response to the third task execution engine being deployed in the third cloud environment, at least one data processing subtask executed by the third task execution engine is determined from a plurality of data processing subtasks as a third data processing subtask.

[0190] Illustratively, the third task execution engine is any computing engine to be analyzed. For example, a task execution engine for executing a data processing subtask is determined based on the data processing subtask. Taking the task execution engine as the third task execution engine as an example, the deployment location of the third task execution engine is determined, that is, the cloud environment in which the third task execution engine is located within the at least one cloud environment.

[0191] Among them, multiple computing engines run in at least one cloud environment, at least one cloud environment includes a third cloud environment that is communicatively connected to the first cloud environment, and no components for overall management of the computing engines are deployed in the third cloud environment.

[0192] That is, the third cloud environment is a cloud environment different from the first cloud environment. The third cloud environment does not deploy components for coordinating and managing computing engines, and is also different from the second cloud environment.

[0193] In some embodiments, when it is determined that the third task execution engine is deployed in the third cloud environment, at least one data processing subtask executed by the third task execution engine is determined from the plurality of data processing subtasks as the third data processing subtask.

[0194] The third data processing subtask is used to represent a data processing subtask processed by a computing engine deployed in a third cloud environment.

[0195] Schematically, the third task execution engine is a task execution engine determined when executing data processing subtask A3, and data processing subtask A3 is used as the above-mentioned third data processing subtask; in addition, other data processing subtasks executed by the third task execution engine can be determined based on the third task execution engine and the execution correspondence. The other data processing subtasks are also data processing subtasks executed by the third task execution engine by calling the data source in the third cloud environment, and the other data processing subtasks are used as the third data processing subtasks.

[0196] That is, the third data processing subtask is implemented as at least one data processing subtask processed by a computing engine deployed in the third cloud environment.

[0197] Step 442: Send the third data processing subtask to the third task execution engine.

[0198] Schematically, based on the fact that no components for managing computing engines are deployed in the third cloud environment, multiple third task execution engines are registered on the first component through direct registration. After the first component determines the third task execution engine and the corresponding third data processing subtask, the first component can send the third data processing subtask to the third task execution engine.

[0199] As shown in Figure 5, taking cloud environment 510 as the third cloud environment as an example, no component (Super SQL) is deployed in the cloud environment 510, and multiple data sources therein (such as the Postgre data source and Hudi data source shown in the figure) are indirectly registered on the first component 521 by registering in the computing engine presto; after the first component 521 is split to obtain multiple data processing subtasks, for the third task execution engine 511 corresponding to any data processing subtask (such as: the third task execution engine is implemented as Presto, etc.), when the third task execution engine 511 is deployed in the third cloud environment, at least one data processing subtask executed by the third task execution engine 511 is determined from the multiple data processing subtasks as the third data processing subtask; the first component 521 in the cloud environment 520 sends the split third data processing subtask to the corresponding third task execution engine 511.

[0200] To sum up, data processing tasks are split with the help of multiple data sources to determine the data processing subtasks executed by different computing engines based on the computing engine's call to the data source. By splitting the data processing tasks, the way of executing data processing tasks is simplified, and different computing engines are fully utilized to perform more targeted execution of data processing subtasks, thereby improving task execution efficiency.

[0201] In an embodiment of the present application, it is introduced that when a first component deployed in a first cloud environment assigns a data processing subtask to a task execution engine, the method of sending the subtask is determined by the deployment location of the task execution engine. While using a management component (such as a first component, a second component, etc.) to manage the computing engines in the cloud environment configured by it, according to the registration relationship between the first component and multiple computing engines, as well as the deployment locations corresponding to the multiple computing engines, a method that is more suitable for overall management of the computing engines is selected, and the data processing subtasks obtained by splitting the first component can be accurately sent to the task execution engine, thereby achieving the purpose of distributing multiple data processing subtasks, thereby facilitating multiple computing engines to perform more targeted execution of the received data processing subtasks.

[0202] In one embodiment, multiple computing engines are deployed in at least one cloud environment. Different registration methods are used to register the multiple computing engines and multiple data sources with the first component, depending on the cloud environment in which the computing engines and data sources are located. As shown in FIG. 6 , the embodiment shown in FIG. 2 further includes steps 610 to 630.

[0203] Step 610: Determine first deployment locations corresponding to the plurality of computing engines and second deployment locations corresponding to the plurality of data sources.

[0204] The first deployment location is used to represent the cloud environment where the computing engine is located, and the second deployment location is used to represent the cloud environment where the data source is located.

[0205] Schematically, multiple computing engines are deployed in at least one cloud environment. The computing engines each have a corresponding cloud environment. The cloud environments in which the multiple computing engines are located are determined to obtain first deployment locations corresponding to the computing engines.

[0206] Two computing engines located in different cloud environments may have the same data processing capabilities or different data processing capabilities.

[0207] Schematically, multiple data sources are deployed in at least one cloud environment, each data source has a corresponding cloud environment, and the cloud environments in which the multiple data sources are located are determined to obtain second deployment locations corresponding to the data sources.

[0208] Two data sources located in different cloud environments may be the same data source (eg, store the same data) or different data sources (eg, store different data).

[0209] Step 620: Register the multiple computing engines on the first component based on the first deployment locations corresponding to the multiple computing engines.

[0210] Schematically, first deployment locations corresponding to the plurality of computing engines are determined, and then, based on the differences represented by the first deployment locations, corresponding registration methods are used to register the different computing engines on the first component.

[0211] In one embodiment, the first component is deployed within a first cloud environment.

[0212] Illustratively, the first component can comprehensively manage the computing engines in the first cloud environment.

[0213] In some embodiments, in response to the first computing engine being located in the first cloud environment, the first computing engine is registered with the first component using a database connection method.

[0214] The first computing engine is any one of the multiple computing engines; a first deployment location corresponding to the first computing engine is determined, and if the first deployment location indicates that the first computing engine is located in the first cloud environment, the first computing engine is registered to the first component using a database connection method.

[0215] Illustratively, the first computing engine is registered on the first component using a Java Database Connectivity (JDBC) method.

[0216] As shown in Figure 7, it is an architectural diagram for registering a computing engine and a data source to a first component, which includes three cloud environments, namely cloud environment 710, cloud environment 720, and cloud environment 730. Taking cloud environment 710 as the first cloud environment as an example, a first component 711 is deployed in cloud environment 720. To facilitate the representation of the registration relationship, the first component 711 is represented separately from the computing engine and data source deployed in cloud environment 710.

[0217] The computing engine Spark and the computing engine Presto are computing engines deployed in the cloud environment 710 , so the first computing engine is registered to the second component 721 using a database connection method.

[0218] In one embodiment, the first component is deployed in a first cloud environment, and at least one cloud environment includes a second cloud environment communicatively connected to the first cloud environment, and the second component is deployed in the second cloud environment.

[0219] Illustratively, the second component can comprehensively manage the computing engines in the second cloud environment.

[0220] In some embodiments, in response to the second computing engine being located in the second cloud environment, the second computing engine is registered with the second component using a database connection.

[0221] The second computing engine is any one of the multiple computing engines; determining a first deployment location corresponding to the second computing engine, if the first deployment location indicates that the second computing engine is located in the second cloud environment, registering the second computing engine to the second component using a database connection method.

[0222] Illustratively, the second computing engine is registered on the second component using the JDBC method.

[0223] In some embodiments, a component registration relationship is established between the second component and the first component.

[0224] The component registration relationship includes indirectly registering the second computing engine registered in the second component to the first component.

[0225] Schematically, considering that a second component is deployed in the second cloud environment, the second component is used to coordinate the management of the computing engine in the second cloud environment. The second computing engine is registered to the second component, so that the second component can manage the second computing engine while conducting more targeted data communication with the first cloud environment through the first component.

[0226] A component registration relationship is established between the second component and the first component using the JDBC method. For example, the first component and the second component involved in the calculation are registered with each other using the JDBC method.

[0227] As shown in Figure 7, taking cloud environment 720 as the second cloud environment as an example, a second component 721 is deployed therein; based on the fact that the computing engine StarRocks and the computing engine Presto are the second computing engines deployed in the cloud environment 720, the second computing engine is registered to the second component 721 using a database connection method; in addition, a component registration relationship between the second component 721 and the first component 711 is established using the JDBC method, so that the second computing engine registered on the second component 721 is indirectly registered on the first component 711.

[0228] In one embodiment, the first component is deployed in a first cloud environment, and at least one cloud environment includes a third cloud environment communicatively connected to the first cloud environment, and the third cloud environment is not deployed with a component for managing a computing engine.

[0229] In some embodiments, in response to the third computing engine being located in the third cloud environment, the third computing engine is registered with the first component using a database connection method.

[0230] The third computing engine is any one of the multiple computing engines; a first deployment location corresponding to the third computing engine is determined; if the first deployment location indicates that the third computing engine is located in a third cloud environment, considering that the third cloud environment has a cross-environment relationship with the first cloud environment, but that no components for coordinating computing engines are deployed in the third cloud environment, the computing engine is registered with the first component as a special data source. For example, the third computing engine is registered with the first component using a JDBC method.

[0231] As shown in Figure 7, taking cloud environment 730 as the third cloud environment as an example, no components are deployed therein; based on the fact that the computing engine Presto therein is the third computing engine deployed in the cloud environment 730, the third computing engine is regarded as a special data source, and the third computing engine is registered to the first component 711 using a database connection method.

[0232] Step 630: Register the multiple data sources on the first component based on the second deployment locations corresponding to the multiple data sources.

[0233] Illustratively, second deployment locations corresponding to the plurality of data sources are determined, and then, based on the differences represented by the second deployment locations, corresponding registration methods are used to register the different data sources on the first component.

[0234] In one embodiment, the first component is deployed within a first cloud environment.

[0235] Illustratively, the first component implements overall management of the computing engines in the first cloud environment by managing the data sources called by the computing engines in the first cloud environment.

[0236] In some embodiments, in response to the first data source being located in the first cloud environment, the first data source is registered on the first component using a database connection method.

[0237] The first data source is any one of the multiple data sources; a second deployment location corresponding to the first data source is determined; if the second deployment location indicates that the first data source is located in the first cloud environment, the first data source is registered to the first component using a database connection method.

[0238] Illustratively, the first data source is registered on the first component using a JDBC method.

[0239] As shown in Figure 7, taking cloud environment 710 as the first cloud environment, a first component 711 is deployed in cloud environment 710. To facilitate the representation of the registration relationship, first component 711 is represented separately from the computing engine and data source deployed in cloud environment 710. Because the data sources MySQL, Iceberg, and Hive are deployed in cloud environment 710, the first data source is registered with first component 711 using JDBC.

[0240] In one embodiment, the first component is deployed in a first cloud environment, and at least one of the cloud environments includes a second cloud environment communicatively connected to the first cloud environment, wherein the second component is deployed in the second cloud environment;

[0241] Illustratively, the first component implements overall management of the computing engine in the second cloud environment by managing the data source called by the computing engine in the second cloud environment.

[0242] In some embodiments, in response to the second data source being located in the second cloud environment, the second data source is registered on the second component using a database connection method.

[0243] The second data source is any one of the multiple data sources; a second deployment location corresponding to the second data source is determined; if the second deployment location indicates that the second data source is located in the second cloud environment, the second data source is registered to the second component using a database connection method.

[0244] Illustratively, the second data source is registered on the second component using a JDBC method.

[0245] In some embodiments, a component registration relationship is established between the second component and the first component.

[0246] The component registration relationship also includes indirectly registering the second data source registered in the second component with the first component. That is, the component registration relationship between the second component and the first component not only represents the indirect registration of the second computing engine registered in the second component with the first component, but also represents the indirect registration of the second data source registered in the second component with the first component.

[0247] Schematically, considering that a second component is deployed in the second cloud environment, the second component realizes the overall management of the computing engine therein by managing the data source called by the computing engine in the second cloud environment, and registers the second data source to the second component, so that the second component can manage the second computing engine and the situation of the second computing engine calling the data source while conducting more targeted data communication with the first cloud environment through the first component.

[0248] A component registration relationship is established between the second component and the first component using the JDBC method. For example, the first component and the second component involved in the calculation are registered with each other using the JDBC method.

[0249] As shown in Figure 7, taking cloud environment 720 as the second cloud environment as an example, a second component 721 is deployed therein; based on the fact that the data source Hudi and the data source Hive are the second data sources deployed in the cloud environment 720, the second data source is registered to the second component 721 using a database connection method; in addition, a component registration relationship between the second component 721 and the first component 711 is established using the JDBC method, so that the second data source registered on the second component 721 is indirectly registered on the first component 711.

[0250] In one embodiment, the first component is deployed in a first cloud environment, and at least one cloud environment includes a third cloud environment communicatively connected to the first cloud environment, and the third cloud environment is not deployed with a component for managing a data source.

[0251] In some embodiments, in response to the third data source being located in the third cloud environment, a data source registration relationship is established between the third data source and the third computing engine.

[0252] As shown in Figure 7, taking cloud environment 730 as the third cloud environment as an example, components such as the first component 711 or the second component 721 are not deployed therein; the computing engine Presto is deployed in the cloud environment 730 as the third computing engine, and the data source Hudi and the data source PostgreSQL are data sources deployed in the cloud environment 730. Therefore, the third data source can be registered on the second computing engine using a database connection method (for example, registering the data source Hudi and the data source PostgreSQL on the computing engine Presto), thereby establishing a data source registration relationship between the third data source and the third computing engine.

[0253] The data source registration relationship is used to indirectly register the third data source registered on the third computing engine to the first component.

[0254] Illustratively, based on the registration relationship between the third computing engine and the first component, the third data source registered on the third computing engine is indirectly registered on the first component.

[0255] To sum up, data processing tasks are split with the help of multiple data sources to determine the data processing subtasks executed by different computing engines based on the computing engine's call to the data source. By splitting the data processing tasks, the way of executing data processing tasks is simplified, and different computing engines are fully utilized to perform more targeted execution of data processing subtasks, thereby improving task execution efficiency.

[0256] In an embodiment of the present application, a method for registering multiple data sources and multiple computing engines on a first component is introduced. Through the first deployment locations corresponding to the multiple computing engines, the multiple computing engines are registered using the registration method corresponding to the first deployment location; through the second deployment locations corresponding to the multiple data sources, the multiple data sources are registered using the registration method corresponding to the second deployment location. The registration process enables the first component to obtain metadata corresponding to the multiple data sources, and also facilitates the first component to know the calling status of the multiple computing engines calling the data sources respectively. Furthermore, based on the registration process, the first component can perform a more accurate and detailed task splitting process for the data processing tasks, so as to obtain data processing subtasks that are more suitable for the computing engine to call the data source for processing, so as to improve the task execution efficiency.

[0257] In one embodiment, multiple computing engines process the received data processing subtasks to obtain data processing results. Based on the multiple data processing results, a task execution method is generated, and the task execution method is sent to the receiving client that issued the data processing task. As shown in FIG8 , step 240 shown in FIG2 is followed by steps 810 to 830.

[0258] Step 810 : In response to the first component meeting a preset result acquisition condition, the subtask execution results corresponding to the plurality of computing engines are acquired.

[0259] Among them, the preset result acquisition condition is used to represent the pre-set result acquisition condition, which is used to obtain the subtask execution result after the computing engine completes the execution of the received data processing subtask; the subtask execution result is the processing result obtained after the computing engine executes the data processing subtask.

[0260] For example: data processing subtask A1 is to filter data that meets condition a under data source M. Then, the data source M is called by calling the calculation engine 1 of the data source to query multiple data that meet condition a, and the multiple data are used as the subtask execution results corresponding to the data processing subtask A1.

[0261] After multiple computing engines process the received data processing subtasks respectively and obtain subtask execution results corresponding to the data processing subtasks, when the first component reaches the preset result acquisition condition, the first component analyzes the multiple subtask execution results.

[0262] In one embodiment, the subtask data volume corresponding to the computing engine is estimated based on the data source called by the computing engine when executing the data processing subtask.

[0263] Schematically, the subtask data volume is used to represent the data scale of the subtask execution result.

[0264] For example, computing engine 1 receives data processing subtask A1. When executing data processing subtask A1, it needs to call data source a. Based on the data size of data source a or the data size of the data used in data source a, it estimates the data size occupied by the subtask execution result obtained after processing data processing subtask A1 with the help of the data source, that is, it estimates the subtask data volume corresponding to data processing subtask A1.

[0265] Alternatively, computing engine 1 receives data processing subtask A1 and data processing subtask A2. When executing data processing subtask A1, it needs to call data source a, and when executing data processing subtask A2, it needs to call data source b. Based on data source a and data source b, it estimates the sum of the data sizes occupied by the subtask execution results obtained after processing data processing subtask A1 and data processing subtask A2 with the help of the data sources, that is, it estimates the amount of subtask data corresponding to both data processing subtask A1 and data processing subtask A2.

[0266] Illustratively, the subtask data volumes corresponding to the multiple computing engines are determined based on the above method.

[0267] In one embodiment, based on the subtask data volumes respectively corresponding to the multiple computing engines and the management components respectively corresponding to the multiple computing engines, the subtask data results respectively counted by the multiple management components are determined.

[0268] The management component is a component for managing a computing engine in a cloud environment, such as the first component in the first cloud environment or the second component in the second cloud environment. The plurality of management components includes the first component.

[0269] Illustratively, multiple computing engines are distributed in at least one cloud environment. Some of the computing engines may be located in cloud environments where management components are deployed (e.g., in the first cloud environment or the second cloud environment mentioned above), while some of the computing engines may not be located in cloud environments where management components are deployed (e.g., in the third cloud environment mentioned above).

[0270] The subtask data volumes corresponding to the multiple computing engines are determined, and the management components corresponding to the multiple computing engines are determined according to the cloud environments in which the multiple computing engines are located.

[0271] Schematically, computing engine 1 is in cloud environment A where management component a is deployed, then the management component corresponding to computing engine 1 is management component a; computing engine 2 is in cloud environment C where no management component is deployed, then the management component corresponding to computing engine 1 is "none", that is, there may be computing engines among multiple computing engines that do not correspond to management components.

[0272] Determine the subtask data results counted by multiple management components.

[0273] Schematically, based on the correspondence between multiple computing engines and multiple management components, at least one computing engine corresponding to each management component is determined, the amount of subtask data corresponding to at least one computing engine is determined, the sum of the subtask data amounts is used as the subtask data result corresponding to the management component, and the subtask data results corresponding to multiple management components are determined.

[0274] In one embodiment, in response to the subtask data result counted by the first component being the largest, subtask execution results corresponding to multiple computing engines are obtained.

[0275] Among them, the maximum subtask data result is the preset result acquisition condition.

[0276] Schematically, multiple management components include a first component. If the subtask execution result counted by the first component is the largest, which means that the data volume of the subtask execution result after the computing engine managed by the first component executes the data processing subtask accounts for a large proportion, then the first component will be used as a component for summarizing the execution results of other subtasks, so that when other management components send the subtask execution results to the first component, the amount of data transmitted in the network is smaller.

[0277] Step 820: Summarize the execution results of multiple subtasks to obtain a task execution result.

[0278] Illustratively, if the first component receives multiple subtask execution results, the multiple subtask execution results are aggregated to obtain a task execution result corresponding to the data processing task.

[0279] Step 830: Send the task execution result to the receiving client.

[0280] Schematically, after the first component summarizes the task execution results corresponding to the data processing task, it sends the task execution results to the receiving client, which is the client that publishes the data processing task, so that the task publishing object is aware of the processing status of the data processing task.

[0281] To sum up, data processing tasks are split with the help of multiple data sources to determine the data processing subtasks executed by different computing engines based on the computing engine's call to the data source. By splitting the data processing tasks, the way of executing data processing tasks is simplified, and different computing engines are fully utilized to perform more targeted execution of data processing subtasks, thereby improving task execution efficiency.

[0282] In one embodiment, the above-mentioned task execution method is applied to a multi-cloud environment (an environment composed of multiple cloud environments), which is called "a hybrid cloud computing method and architecture across computing engines". The above-mentioned task execution method can be applied in a hybrid cloud environment for big data computing, and the data routing layer is realized by adapting the distributed file system.

[0283] Schematically, as shown in FIG7 above, the overall architecture of the task execution method is introduced, which includes three cloud environments, namely cloud environment 710, cloud environment 720 and cloud environment 730. Taking cloud environment 710 as the first cloud environment as an example, a first component 711 is deployed in cloud environment 710. To facilitate the representation of the registration relationship, the first component 711 is represented separately from the computing engine and data source deployed in cloud environment 710.

[0284] The management component is further configured with a data routing unit and a metadata statistics unit, such as: the data routing unit and metadata statistics unit in the first component 711; and the data routing unit and metadata statistics unit in the second component 721.

[0285] The data routing unit is primarily used to connect to distributed file systems located on different cloud platforms (cloud environments). During data transmission, the data routing module in the first component 711 of the cloud environment 710 is responsible for processing and reading intermediate data results from the remote distributed file system. Simultaneously, the local data routing module (such as the second component 721) is responsible for receiving the intermediate data results transmitted from the remote end and writing the data to a designated directory in the local distributed file system.

[0286] The metadata statistics unit is primarily used to integrate metadata database tables (metadata corresponding to multiple data sources) and statistical information from management components deployed in multiple cloud environments to form a complete metadata view. This allows the first component to split data processing tasks into multiple data processing subtasks based on the metadata database tables. Furthermore, the metadata statistics can be used to optimize execution plans and estimate subtask execution data.

[0287] Schematically, as shown in Figure 7, a multi-cloud environment participating in joint analysis will deploy multiple management components (SuperSQL services). When multiple management components register with each other as engines, they have equal status and can submit cross-cloud joint analysis tasks (i.e., jointly execute data processing tasks in multiple cloud environments).

[0288] Before executing data processing tasks, it is necessary to connect computing resources and data across multiple clouds. The computing engine (mainly the computing engine deployed in the first cloud environment, and the SuperSQL service itself can also be regarded as a computing engine) and the data source (mainly local) are connected to the SuperSQL service through a registration mechanism, so that the SuperSQL service has a unified metadata view for splitting data processing tasks, executing data processing subtasks, and summarizing subtask execution results.

[0289] Compared to previous approaches that relied solely on data source registration, achieving unique identification in cross-cloud scenarios is more difficult, and network connectivity is also difficult to guarantee. Compute engine registration not only improves the management efficiency of metadata corresponding to data sources, but also supports access to diverse computing resources to meet different task types and computing requirements. This not only adapts to cross-cloud scenarios but also accommodates various network environments, improving system flexibility and stability.

[0290] (1) The SuperSQL services involved in the calculation register with each other via JDBC. For example, the first component in the cloud environment 710 and the second component in the cloud environment 720 register with each other via JDBC.

[0291] (2) If a SuperSQL service exists within the cloud environment, the computing engine and data source within the cloud environment are registered with the SuperSQL service via JDBC. For example, the computing engine and data source deployed within cloud environment 710 are registered with first component 711 via JDBC; the computing engine and data source deployed within cloud environment 720 are registered with second component 721 via JDBC.

[0292] (3) If the SuperSQL service does not exist within the cloud environment, but a computing engine exists and a corresponding data source is configured within it, the computing engine needs to be registered as a special data source with the remote SuperSQL service via JDBC. Register the computing engine and data source within the cloud environment 730 with the first component.

[0293] Schematically, through the above registration process, the first component (SuperSQL service) can obtain the engine information corresponding to the computing engine through the standard JDBC interface provided by the computing engine, so as to know the data source called by the computing engine and determine the calling correspondence between the data source and the computing engine; the first component can also obtain the metadata corresponding to the data source through the standard JDBC interface provided by the data source to know the attribute information of the data source, such as: the amount of data stored therein, the data type, etc., and then summarize and construct a unified metadata view (which characterizes the relationship between the data source and the computing engine that calls the data source). Among them, the computing engine in the above (3) is the data source of special data. Since the data routing module can only be used to read the intermediate data results in the remote distributed file system, the result data can only be transmitted through JDBC or the interface provided by the computing engine, and cannot be transmitted through data routing.

[0294] In one embodiment, as shown in FIG5 , it is a schematic diagram of an architecture for distributing multiple data processing subtasks to a task execution engine, which includes three cloud environments, namely, cloud environment 510 , cloud environment 520 , and cloud environment 530 .

[0295] Taking the cloud environment 520 as the first cloud environment as an example, a first component 521 is deployed in the cloud environment 520, and the task execution method is executed with the help of the first component 521.

[0296] (1) Using an object to submit a data processing task (SQL task) to the first component (SuperSQL service) of the cloud environment 520, which can be called a submitter;

[0297] (2) After parsing and verifying the data processing task, the first component splits it among multiple computing engines or management components within the cloud environment based on metadata. This task splitting also includes operator pushdown, which is used to filter and reduce network data transmission traffic. The SQL syntax submitted by the user is the unified SuperSQL syntax. After splitting the data processing subtasks, the first component converts the dialect of the data processing subtask into the dialect of the corresponding computing engine.

[0298] (3) After receiving the data processing subtask, the management component and the computing engine will have different processing methods, as shown below:

[0299] 1. The computing engine directly executes the received data processing subtask and returns the ResultSet to the first component through the standard JDBC interface or the interface provided by the computing engine;

[0300] 2. After receiving the data processing subtask, other management components outside the first component (e.g., the second component) perform the same processing as the first component, parsing, validating, and splitting the data based on metadata. Because the data processing subtasks are already split, this splitting does not change the task content. For example, when the second component receives a data processing subtask that requires invoking at least two data sources, it can copy the subtask into two subtasks with the same content.

[0301] The second component then selects a task execution engine for the data processing subtask based on statistical information from the second component, including database and table information, query relationships, and metadata. Once the task execution engine is determined, the second component translates the syntax of the data processing subtask into the engine's dialect. The task execution engine then processes the subtask and writes the resulting data as a file to the cloud environment's distributed file system. The data is then output to the file system on the aggregation end according to the data routing algorithm.

[0302] 3. If there are data processing subtasks that need to be processed by the first component, the processing steps refer to the processing method of the second component mentioned above.

[0303] Illustratively, when all data processing subtasks have completed and obtained their results, a management component (e.g., the first component) is determined to handle these subtask results. This component is referred to as the aggregation end. The distributed file system within the aggregation end then contains multiple file directories, equal to the number of data processing subtasks. The aggregation end then creates temporary tables based on these file directories, performs the final aggregation calculation, and obtains the task execution results. This result is then returned to the user via the JDBC interface of the first component.

[0304] In one embodiment, as shown in FIG9 , a data routing algorithm is described, which includes the following steps 910 to 980 .

[0305] Step 910: Estimate the size of result data corresponding to each of the plurality of management components.

[0306] Illustratively, the first component estimates the result data size of the data processing subtasks on the remaining management components based on the statistical information of the metadata (RBO or CBO), that is, the above-mentioned subtask data results.

[0307] Step 920: sort the result data sizes corresponding to the multiple management components.

[0308] Schematically, multiple management components are sorted according to the estimated result data size, and the largest management component is selected as the aggregation end.

[0309] If the metadata statistics of a management component are insufficient and the size of the result data cannot be estimated, to ensure stability, the result data of the management component will be written to the local distributed file system by default before being transmitted to the aggregation end. In addition, the management component cannot be used as the end that performs the aggregation task.

[0310] Step 930: Use the management component with the largest result data as the aggregation end.

[0311] Illustratively, when the result data (subtask data result) of the first component among multiple management components is the largest, the first component is used as the management component for summarizing the execution results of multiple subtasks.

[0312] Step 940: Determine whether the result data of the non-aggregation end is greater than the transmission threshold.

[0313] Illustratively, taking the first component as a management component that aggregates the execution results of multiple subtasks as an example, result data corresponding to other management components except the first component among the multiple management components are determined, and compared with a preset transmission threshold.

[0314] The preset transmission threshold is 500M, and the result data corresponding to other management components are compared with 500MB.

[0315] If the result data of the non-aggregation end is greater than the transmission threshold, execute the following step 950; if the result data of the non-aggregation end is not greater than the transmission threshold, execute the following step 980.

[0316] Step 950: Determine whether the result data of the non-aggregation end exceeds the compression threshold.

[0317] Schematically, if the result data of the non-aggregation end is greater than the transmission threshold, the result data of the management component marked needs to be written into the local distributed file system first (the result storage path and file format are specified by the data routing module of the submission end and passed to the execution end as parameters), and then transmitted to the aggregation end (such as the first component) via HTTP.

[0318] In addition, it is determined whether the result data of the non-aggregation end exceeds a preset compression threshold. If the preset compression threshold is 1GB, the result data corresponding to the management component is compared with 1GB.

[0319] If the result data of the non-aggregation end is greater than the compression threshold, execute the following step 960; if the result data of the non-aggregation end is not greater than the compression threshold, execute the following step 970.

[0320] Step 960: compress and transmit to the aggregation end.

[0321] For example, if the result data on the non-aggregator side exceeds the compression threshold, the result data must be compressed first, transmitted to the aggregation side via HTTP, and then decompressed. The compression algorithm is specified by the submitter and passed to the execution and aggregation sides when the task is submitted.

[0322] Step 970: directly transmit to the aggregation end.

[0323] Indicatively, if the result data at the non-aggregation end is not greater than the compression threshold, the calculated result data does not need to be compressed and is directly transmitted to the aggregation end via HTTP.

[0324] Step 980: Transmit to the aggregation end via JDBC ResultSet.

[0325] Indicatively, if the result data on the non-aggregation end is not greater than the transmission threshold, it is transmitted to the aggregation end through the JDBC ResultSet.

[0326] It should be noted that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0327] In one embodiment, the above-mentioned task execution method adds a data routing module to the management component (such as the first component, the second component, etc.) to complete data transmission across the cloud environment and the computing engine, wherein the data routing module is deployed separately and provides an HTTP interface for the management component to call.

[0328] The following sections describe the configuration parameters for enabling management components.

[0329] (1) supersql.datarouter.enabled = true; This configuration parameter is a global switch and is associated with the following configuration parameters, as shown in (2) to (5).

[0330] (2) supersql.datarouter.result.dir = / tmp / data; This configuration parameter is used to indicate the data storage location of the data processing subtask;

[0331] (3) supersql.datarouter.result.size = 500MB; This configuration parameter is used to indicate the data transmission method boundary of the data processing subtask. If it is less than 500MB, the data is transmitted via JDBC. If it is larger than 500MB, the data is transmitted via HTTP. The 500MB here is a schematic illustration of the transmission threshold and is not limited here.

[0332] (4) supersql.datarouter.compression.result.enable = true; This configuration parameter is used to indicate whether the data of the data processing subtask is compressed before transmission;

[0333] (5)supersql.datarouter.encryption.result.enable=true; This configuration parameter is used to indicate whether the data of the data processing subtask is encrypted before transmission.

[0334] In addition, multiple configuration parameters can be configured on the management component, which are not listed here one by one. For example, the product side needs to add a data source of the corresponding engine type (such as SuperSQL, Presto, Spark), etc. This embodiment of the application does not limit this.

[0335] To sum up, data processing tasks are split with the help of multiple data sources to determine the data processing subtasks executed by different computing engines based on the computing engine's call to the data source. By splitting the data processing tasks, the way of executing data processing tasks is simplified, and different computing engines are fully utilized to perform more targeted execution of data processing subtasks, thereby improving task execution efficiency.

[0336] In an embodiment of the present application, the data processing method can be applied in a hybrid cloud scenario to realize joint computing and analysis across computing engines, making multi-cloud joint computing possible; it is also possible to use multiple computing engines within the data center based on the characteristics of the computing engines and the data scale to speed up the joint analysis of multiple tables and reduce the waste of engine computing resources; it is also possible to use the operator push-down optimization method in the task splitting process to achieve filtering and reduce the traffic of network data transmission; in addition, it is also possible to use compression and other methods during the data transmission process to reduce cross-cloud network traffic while reducing the cost of joint analysis.

[0337] FIG10 is a structural block diagram of a task execution device provided by an exemplary embodiment of the present application. As shown in FIG10 , the device includes the following parts:

[0338] The task receiving module 1010 is used to receive a data processing task, wherein the data processing task is a task to be collaboratively processed by multiple computing engines, and the first component has registered therewith the multiple computing engines and multiple data sources called by the computing engines;

[0339] A task splitting module 1020 is configured to split the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks, and the computing engine is configured to call the data source to execute at least one data processing subtask;

[0340] a relationship determination module 1030, configured to determine an execution correspondence relationship between the plurality of data processing subtasks and the plurality of computing engines;

[0341] The task distribution module 1040 is configured to distribute the plurality of data processing subtasks to a task execution engine among the plurality of computing engines based on the execution correspondence, wherein the task execution engine is a computing engine configured to execute the data processing subtasks.

[0342] In one embodiment, the task splitting module 1020 is also used to determine the calling correspondence between the multiple data sources and the multiple computing engines registered on the first component, and the calling correspondence is used to characterize the relationship between the data source being called by the computing engine; based on the multiple data sources registered on the first component and the calling correspondence, the data processing task is split to obtain the multiple data processing subtasks.

[0343] In one embodiment, the task splitting module 1020 is also used to determine the task processing part implemented by at least one data source under the same computing engine when executing the data processing task based on the correspondence between the multiple data sources registered on the first component and the call; and split the data processing task based on the task processing parts corresponding to the multiple computing engines to obtain the multiple data processing subtasks.

[0344] In one embodiment, the task splitting module 1020 is also used to obtain metadata corresponding to the multiple data sources, where the metadata is used to characterize attribute information corresponding to the data sources; and split the data processing task based on the metadata to obtain the multiple data processing subtasks.

[0345] In one embodiment, the task splitting module 1020 is also used to send the data processing task to at least one data source among the multiple data sources, and the at least one data source is used to filter the data stored in the data source through the data processing task to obtain task data; and split the data processing task based on the task data to obtain the multiple data processing subtasks.

[0346] In one embodiment, the first component is deployed in a first cloud environment;

[0347] The task distribution module 1040 is also used to determine the task execution engines corresponding to the multiple data processing subtasks based on the execution correspondence; and distribute the multiple data processing subtasks to the corresponding task execution engines based on the deployment relationship between the deployment location of the task execution engine and the first cloud environment.

[0348] In one embodiment, the task distribution module 1040 is also used to determine at least one data processing subtask executed by the first task execution engine as the first data processing subtask from the multiple data processing subtasks in response to the first task execution engine being deployed in the first cloud environment; send the first data processing subtask to the first task execution engine, and distribute the multiple data processing subtasks to the corresponding task execution engines.

[0349] In one embodiment, the plurality of computing engines are run in at least one cloud environment, the at least one cloud environment includes a second cloud environment communicatively connected to the first cloud environment, and a second component is deployed in the second cloud environment;

[0350] The task distribution module 1040 is also used to determine at least one data processing subtask executed by the second task execution engine from the multiple data processing subtasks as a second data processing subtask in response to the second task execution engine being deployed in the second cloud environment; send the second data processing subtask to the second component, and the second component is used to send the second data processing subtask to the second task execution engine; and distribute the multiple data processing subtasks to the corresponding task execution engines.

[0351] In one embodiment, the multiple computing engines are run in at least one cloud environment, the at least one cloud environment includes a third cloud environment communicatively connected to the first cloud environment, and the third cloud environment does not have a component for coordinating and managing the computing engines deployed therein;

[0352] The task distribution module 1040 is also used to determine at least one data processing subtask executed by the third task execution engine as a third data processing subtask from the multiple data processing subtasks in response to the third task execution engine being deployed in the third cloud environment; send the third data processing subtask to the third task execution engine, and distribute the multiple data processing subtasks to the corresponding task execution engines.

[0353] In one embodiment, as shown in FIG11 , the plurality of computing engines are deployed in at least one cloud environment;

[0354] The device further comprises:

[0355] Registration module 1050 is used to determine the first deployment locations corresponding to the multiple computing engines and the second deployment locations corresponding to the multiple data sources, the first deployment locations are used to represent the cloud environment where the computing engines are located, and the second deployment locations are used to represent the cloud environment where the data sources are located; based on the first deployment locations corresponding to the multiple computing engines, the multiple computing engines are registered on the first component; based on the second deployment locations corresponding to the multiple data sources, the multiple data sources are registered on the first component.

[0356] In one embodiment, the first component is deployed in a first cloud environment;

[0357] The registration module 1050 is also used to register the first computing engine to the first component using a database connection method in response to the first computing engine being located in the first cloud environment; and to register the first data source to the first component using the database connection method in response to the first data source being located in the first cloud environment.

[0358] In one embodiment, the first component is deployed in a first cloud environment, the at least one cloud environment includes a second cloud environment communicatively connected to the first cloud environment, and the second component is deployed in the second cloud environment;

[0359] The registration module 1050 is also used to register the second computing engine to the second component using a database connection method in response to the second computing engine being located in the second cloud environment; establish a component registration relationship between the second component and the first component, and the component registration relationship includes indirectly registering the second computing engine registered in the second component to the first component; in response to the second data source being located in the second cloud environment, register the second data source to the second component using the database connection method; establish the component registration relationship between the second component and the first component, and the component registration relationship also includes indirectly registering the second data source registered in the second component to the first component.

[0360] In one embodiment, the first component is deployed in a first cloud environment, the at least one cloud environment includes a third cloud environment communicatively connected to the first cloud environment, and the third cloud environment is not deployed with a management component for managing a computing engine;

[0361] The registration module 1050 is also used to register the third computing engine to the first component using a database connection method in response to the third computing engine being located in the third cloud environment; in response to the third data source being located in the third cloud environment, establish a data source registration relationship between the third data source and the third computing engine, and the data source registration relationship is used to indirectly register the third data source registered on the third computing engine to the first component.

[0362] In one embodiment, as shown in FIG11 , the apparatus further comprises:

[0363] The result sending module 1060 is used to obtain the subtask execution results corresponding to the multiple computing engines in response to the first component reaching the preset result acquisition condition; summarize the multiple subtask execution results to obtain the task execution result; and send the task execution result to the receiving client, which is the client that publishes the data processing task.

[0364] In one embodiment, the result sending module 1060 is also used to estimate the subtask data volume corresponding to the computing engine based on the data source called by the computing engine when executing the data processing subtask, and the subtask data volume is used to characterize the data scale of the subtask execution result; based on the subtask data volumes corresponding to the multiple computing engines respectively, and the management components corresponding to the multiple computing engines respectively, determine the subtask data results respectively counted by multiple management components, the management component is a component for managing the computing engine in the cloud environment, and the multiple management components include the first component; in response to the subtask data result counted by the first component being the largest, obtain the subtask execution results corresponding to the multiple computing engines respectively, and the maximum subtask data result is the preset result acquisition condition.

[0365] To sum up, data processing tasks are split with the help of multiple data sources to determine the data processing subtasks executed by different computing engines based on the computing engine's call to the data source. By splitting the data processing tasks, the way of executing data processing tasks is simplified, and different computing engines are fully utilized to perform more targeted execution of data processing subtasks, thereby improving task execution efficiency.

[0366] It should be noted that the task execution device provided in the above embodiment is merely an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the task execution device provided in the above embodiment and the task execution method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0367] Figure 12 shows a schematic diagram of the structure of a server device provided by an exemplary embodiment of the present application. The server device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. The server device 1200 also includes a mass storage device 1206 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0368] The mass storage device 1206 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1206 and its associated computer-readable media provide non-volatile storage for the server device 1200. In other words, the mass storage device 1206 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0369] Computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. The system memory 1204 and mass storage device 1206 described above may be collectively referred to as memory.

[0370] According to various embodiments of the present application, the server device 1200 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server device 1200 may be connected to the network 1212 via the network interface unit 1211 connected to the system bus 1205. Alternatively, the network interface unit 1211 may be used to connect to other types of networks or remote computer systems (not shown).

[0371] The memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0372] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the task execution method provided by the above-mentioned method embodiments.

[0373] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the task execution method provided by the above-mentioned method embodiments.

[0374] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the task execution method described in any of the above embodiments.

[0375] The above description is merely an embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A task execution method, characterized in that: The method is performed by a first component, and includes: receiving a data processing task, wherein the data processing task is a task to be collaboratively processed by a plurality of computing engines, wherein the plurality of computing engines and at least one data source respectively called by the plurality of computing engines are registered on the first component; The data processing task is split based on the multiple data sources registered on the first component to obtain multiple data processing subtasks, and the computing engine is used to call the data source to execute at least one data processing subtask; Determining an execution correspondence relationship between the plurality of data processing subtasks and the plurality of computing engines; Based on the execution correspondence, the multiple data processing subtasks are distributed to task execution engines among the multiple computing engines, and the task execution engines are computing engines used to execute data processing subtasks.

2. The method according to claim 1, characterized in that The step of splitting the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks includes: Based on the multiple computing engines registered on the first component, determining a call correspondence relationship between the multiple data sources and the multiple computing engines, wherein the call correspondence relationship is used to characterize a relationship in which the data source is called by the computing engine; Based on the multiple data sources registered on the first component and the call correspondence relationship, the data processing task is split to obtain the multiple data processing subtasks.

3. The method according to claim 2, characterized in that The step of splitting the data processing task based on the multiple data sources registered on the first component and the call correspondence relationship to obtain the multiple data processing subtasks includes: Based on the multiple data sources registered on the first component and the call correspondence relationship, determine a task processing part implemented by at least one data source under the same computing engine when executing the data processing task; The data processing task is split based on the task processing parts respectively corresponding to the multiple computing engines to obtain the multiple data processing subtasks.

4. The method according to any one of claims 1 to 3, characterized in that: The step of splitting the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks includes: Obtain metadata corresponding to the multiple data sources respectively, where the metadata is used to represent attribute information corresponding to the data sources; The data processing task is split based on the metadata to obtain the multiple data processing subtasks.

5. The method according to any one of claims 1 to 3, characterized in that: The step of splitting the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks includes: The data processing task is sent to at least one of the multiple data sources, and the at least one A data source is used to filter the data stored in the data source through the data processing task to obtain task data; The data processing task is split based on the task data to obtain the multiple data processing subtasks.

6. The method according to any one of claims 1 to 3, characterized in that: The first component is deployed in a first cloud environment; The distributing the plurality of data processing subtasks to task execution engines among the plurality of computing engines based on the execution correspondence includes: Based on the execution correspondence, determining the task execution engines corresponding to the plurality of data processing subtasks respectively; Based on the deployment relationship between the deployment location of the task execution engine and the first cloud environment, the multiple data processing subtasks are distributed to the corresponding task execution engine.

7. The method according to claim 6, characterized in that The distributing the plurality of data processing subtasks to corresponding task execution engines based on the deployment relationship between the deployment location of the task execution engine and the first cloud environment includes: In response to a first task execution engine being deployed in the first cloud environment, determining at least one data processing subtask executed by the first task execution engine from the plurality of data processing subtasks as a first data processing subtask; Send the first data processing subtask to the first task execution engine.

8. The method according to claim 6, characterized in that The multiple computing engines are run in at least one cloud environment, the at least one cloud environment includes a second cloud environment that is communicatively connected to the first cloud environment, and a second component is deployed in the second cloud environment; The distributing the plurality of data processing subtasks to corresponding task execution engines based on the deployment relationship between the deployment location of the task execution engine and the first cloud environment includes: In response to a second task execution engine being deployed in the second cloud environment, determining at least one data processing subtask executed by the second task execution engine from the plurality of data processing subtasks as a second data processing subtask; The second data processing subtask is sent to the second component, and the second component is used to send the second data processing subtask to the second task execution engine.

9. The method according to claim 6, characterized in that The multiple computing engines are run in at least one cloud environment, the at least one cloud environment includes a third cloud environment that is communicatively connected to the first cloud environment, and the third cloud environment does not have a component for overall management of the computing engines deployed; The distributing the plurality of data processing subtasks to corresponding task execution engines based on the deployment relationship between the deployment location of the task execution engine and the first cloud environment includes: In response to a third task execution engine being deployed in the third cloud environment, determining at least one data processing subtask executed by the third task execution engine from the plurality of data processing subtasks as a third data processing subtask; The third data processing subtask is sent to the third task execution engine.

10. The method according to any one of claims 1 to 3, characterized in that: The plurality of computing engines are deployed in at least one cloud environment; The method further comprises: Determine first deployment locations corresponding to the multiple computing engines respectively, and second deployment locations corresponding to the multiple data sources respectively, wherein the first deployment locations are used to characterize the cloud environment where the computing engines are located, and the second deployment locations are used to characterize the cloud environment where the data sources are located; Registering the multiple computing engines on the first component based on the first deployment locations respectively corresponding to the multiple computing engines; The multiple data sources are registered on the first component based on the second deployment locations respectively corresponding to the multiple data sources.

11. The method according to claim 10, characterized in that The first component is deployed in a first cloud environment; The registering the plurality of computing engines on the first component based on the first deployment locations respectively corresponding to the plurality of computing engines includes: In response to the first computing engine being located in the first cloud environment, registering the first computing engine to the first component by using a database connection method; The registering the multiple data sources on the first component based on the second deployment locations respectively corresponding to the multiple data sources includes: In response to the first data source being located in the first cloud environment, the first data source is registered on the first component using the database connection method.

12. The method according to claim 10, characterized in that The first component is deployed in a first cloud environment, the at least one cloud environment includes a second cloud environment that is communicatively connected to the first cloud environment, and the second cloud environment has a second component deployed therein; The registering the plurality of computing engines on the first component based on the first deployment locations respectively corresponding to the plurality of computing engines includes: In response to the second computing engine being located in the second cloud environment, registering the second computing engine to the second component by means of a database connection; establishing a component registration relationship between the second component and the first component, wherein the component registration relationship includes indirectly registering the second computing engine registered in the second component to the first component; The registering the multiple data sources on the first component based on the second deployment locations respectively corresponding to the multiple data sources includes: In response to the second data source being located in the second cloud environment, the second data source is registered to the second component using the database connection method; the component registration relationship between the second component and the first component is established, and the component registration relationship also includes indirectly registering the second data source registered in the second component to the first component.

13. The method according to claim 10, characterized in that The first component is deployed in a first cloud environment, the at least one cloud environment includes a third cloud environment that is communicatively connected to the first cloud environment, and the third cloud environment is not deployed with a management component for managing a computing engine; The registering the plurality of computing engines on the first component based on the first deployment locations respectively corresponding to the plurality of computing engines includes: In response to the third computing engine being located in the third cloud environment, registering the third computing engine to the first component by using a database connection method; The registering the multiple data sources on the first component based on the second deployment locations respectively corresponding to the multiple data sources includes: In response to the third data source being located in the third cloud environment, a data source registration relationship is established between the third data source and the third computing engine, wherein the data source registration relationship is used to indirectly register the third data source registered on the third computing engine to the first component.

14. The method according to any one of claims 1 to 3, characterized in that: After distributing the plurality of data processing subtasks to the task execution engines among the plurality of computing engines based on the execution correspondence, the method further includes: In response to the first component meeting a preset result acquisition condition, acquiring subtask execution results respectively corresponding to the plurality of computing engines; Aggregate the execution results of multiple subtasks to obtain the task execution result; The task execution result is sent to a receiving client, where the receiving client is the client that publishes the data processing task.

15. The method according to claim 14, characterized in that In response to the first component meeting a preset result acquisition condition, acquiring subtask execution results respectively corresponding to the plurality of computing engines, comprises: Based on the data source called by the computing engine when executing the data processing subtask, estimate the subtask data volume corresponding to the computing engine, and the subtask data volume is used to characterize the data scale of the subtask execution result; Based on the subtask data amounts respectively corresponding to the multiple computing engines and the management components respectively corresponding to the multiple computing engines, determining the subtask data results respectively counted by the multiple management components, the management component is a component used to manage computing engines in a cloud environment, and the multiple management components include the first component; In response to the subtask data result counted by the first component being the largest, the subtask execution results respectively corresponding to the multiple computing engines are obtained, and the largest subtask data result is the preset result acquisition condition.

16. A task execution device, characterized in that: The device comprises: A task receiving module, used to receive a data processing task, wherein the data processing task is a task to be collaboratively processed by multiple computing engines, and the first component is registered with the multiple computing engines and at least one data source respectively called by the multiple computing engines; A task splitting module, used to split the data processing task based on the multiple data sources registered on the first component to obtain multiple data processing subtasks, and the computing engine is used to call the data source to execute at least one data processing subtask; A relationship determination module is used to determine the relationship between the multiple data processing subtasks and the multiple computing engines. Execute the corresponding relationship; A task distribution module is used to distribute the multiple data processing subtasks to a task execution engine among the multiple computing engines based on the execution correspondence, and the task execution engine is a computing engine used to execute the data processing subtasks.

17. The device according to claim 16, characterized in that The task splitting module is further used to determine the call correspondence relationship between the multiple data sources and the multiple computing engines registered on the first component, wherein the call correspondence relationship is used to characterize the relationship between the data source being called by the computing engine; Based on the multiple data sources registered on the first component and the call correspondence relationship, the data processing task is split to obtain the multiple data processing subtasks.

18. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the task execution method according to any one of claims 1 to 15.

19. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the task execution method as described in any one of claims 1 to 15.

20. A computer program product, characterized in that The computer program product comprises computer instructions, which are loaded and executed by a processor to implement the task execution method according to any one of claims 1 to 15.

Images