Method, system, device and readable storage medium for processing geographically distributed operations

Abstract

The present application provides a method, device, and storage medium for processing geographically distributed jobs. The method includes: receiving a job execution request, which is a request to execute a job and includes job information; generating a job execution plan based at least on the job information, wherein the job execution plan is determined based on the site location of the target data, the data processing capabilities of the site, and the network transmission capabilities between each two sites; and executing the job execution plan so that the target site executes the job according to the job execution plan. The method coordinates computing jobs through a top-level manager context-aware scheduler to find the optimal execution plan for the job, so that the bottom-level sites process the job according to the plan, resulting in a fairer workload distribution between computing jobs. This solves the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular, to a method for processing geographically distributed jobs, a system for processing geographically distributed jobs, a device for processing geographically distributed jobs, and a computer-readable storage medium. Background Art

[0002] Today, an increasing number of applications require access to large amounts of distributed data. Global social networking applications generate vast amounts of data. State-of-the-art parallel computing technologies are ineffective in geographically distributed scenarios, where data is typically stored in several data centers (sites), often located in different regions. Each data center has varying computing power, and each data center requires different raw data to mine. Consequently, the raw data within each data center is unevenly distributed. This means that the computing power of each data center is disproportionate to the raw data it needs to mine, resulting in low data processing efficiency. Summary of the Invention

[0003] The main purpose of this application is to provide a method for processing geographically distributed jobs, a processing system for geographically distributed jobs, a processing device for geographically distributed jobs and a computer-readable storage medium, so as to at least solve the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for processing geographically distributed jobs is provided, comprising: receiving a job execution request, the job execution request being a request for executing a job, the job execution request including job information, the job information including at least one of the following: an algorithm for executing the job, an input shard of the job, and a job description; generating a job execution plan based at least on the job information, wherein the job execution plan is a plan including a target site representing a target site for executing the job, the job execution plan is determined based on the site location where the target data is located, the data processing capability of the site, and the network transmission capability between each two sites, wherein the target site is one or more of all the sites, the target data is the data that needs to be processed to execute the job, the data processing capability of the site is a comprehensive representation of the various computing resources of the site, and the network transmission capability is the network bandwidth between the two sites; executing the job execution plan so that the target site executes the job according to the job execution plan.

[0005] Optionally, a job execution plan is generated at least based on the job information, including: determining the target data based on the target job information, the target job information includes the job information or multiple sub-job information, and the sub-job information is obtained by splitting the job information; obtaining the original site where the target data is located, the data processing capacity of each of the sites, and the network transmission capacity between each two of the sites, the original site is the site where the target data is stored; generating the job execution plan based on the target data, the original site where the target data is located, the data processing capacity of each of the sites, and the network transmission capacity between each two of the sites.

[0006] Optionally, the job execution plan is generated based on the target data, the original site where the target data is located, the data processing capabilities of each site and the network transmission capabilities between every two sites, including: establishing an optimization model based on the target data, the optimization target of the optimization model is the time for processing the target data, the variables of the optimization model include the original site where the target data is located and the preparatory site, the preparatory site is the site that processes the target job information, and the time for processing the target job information includes the sum of the data transmission time between the original site and the preparatory site, the job waiting time of the preparatory site, and the data processing time of the preparatory site; according to the optimization model, determining the feasible solution of the variables corresponding to the minimum optimization target to obtain the job execution plan.

[0007] Optionally, based on the target data, an optimization model is established, including: forming a plurality of nodes based on the target data, the optimization model also including a plurality of nodes and a plurality of edges, the plurality of nodes of the optimization model including data computing nodes, data transmission nodes and job waiting nodes, the data computing nodes representing the data processing flow nodes of the preparatory site, the data transmission nodes representing the data transmission flow nodes from the original site to the preparatory site, the job waiting nodes being the job waiting flow nodes from the time the target data arrives at the preparatory site to the time it is processed, wherein the preparatory site includes the original site; forming a plurality of edges representing the processing flow of the target data, and using the edges to Connect the data transmission node, the job waiting node and the data calculation node for the second time to obtain multiple processing paths of the target data; determine the function of the time taken to calculate the processing path as the optimization target to obtain the optimization model; and / or, based on the optimization model, determine the feasible solution of the variable corresponding to the minimum optimization target to obtain the job execution plan, including: calculating the value of the optimization target corresponding to each of the processing paths; determining the processing path corresponding to the minimum value of the optimization target as the target path, determining the preparatory site corresponding to the target path as the target site, and the original site corresponding to the target path as the target original site to obtain the job execution plan.

[0008] Optionally, when the original site is different from the preparatory site, the value of the optimization target corresponding to each processing path is calculated, including: obtaining the compression factor and throughput of the data calculation node corresponding to the processing path to obtain the compression factor and the first throughput, obtaining the throughput of the data transmission node corresponding to the processing path to obtain the second throughput, and obtaining the waiting time of the target data on the job waiting node corresponding to the processing path, the compression factor being the ratio of the amount of data obtained by the preparatory site processing the target data to the amount of data of the target data, the first throughput being the amount of data processed by the preparatory site per unit time, and the second throughput being the amount of data transmitted from the original site per unit time. The starting site transmits the target data in an amount to the preparatory site, and the waiting time is the time the target data waits from arriving at the preparatory site to being processed; based on the target data and the compression factor, the data processing volume of the target site is determined to be the product of the target data and the compression factor; based on the data processing volume and the first throughput, the processing time is determined to be the ratio of the data processing volume to the first throughput; based on the target data and the second throughput, the transmission time is determined to be the ratio of the target data to the second throughput; based on the target data and the first throughput, the value of the optimization target corresponding to the processing path is determined to be the sum of the waiting time, the transmission time and the processing time.

[0009] Optionally, executing the job execution plan so that the target site executes the job according to the job execution plan includes: when the target site is the original site of the target data, executing the job execution plan so that the target site processes the target data, and the original site is the site that stores the target data; when the target site is not the original site of the target data, executing the job execution plan to control the original site to send the target data to the target site, so that the target site processes the target data.

[0010] Optionally, when the target site is not the original site of the target data, the job execution plan is executed to control the original site to send the target data to the target site, including: controlling the original site to split the target data into multiple data blocks; controlling the original site to send multiple data blocks to the target site.

[0011] Optionally, controlling the original site to split the target data into multiple data blocks includes: obtaining the computing power of the target site; obtaining the network transmission capacity between the original site and the target site; and determining the size of the data block based on the computing power of the original site and the network transmission capacity.

[0012] According to another aspect of the present application, a system for processing geographically distributed jobs is provided, comprising: a top-level management structure for executing any one of the methods for processing geographically distributed jobs; a target site, wherein the target site is used to execute the job according to a job execution plan, wherein the job execution plan is a plan including a target site representing the execution of the job.

[0013] According to another aspect of the present application, a device for processing geographically distributed jobs is provided, comprising: a receiving unit for receiving a job execution request, wherein the job execution request is a request for executing a job, and the job execution request includes job information, and the job information includes at least one of the following: an algorithm for executing the job, an input shard of the job, and a job description; a generating unit for generating a job execution plan based at least on the job information, wherein the job execution plan is a plan including a target site for executing the job, and the job execution plan is determined based on the site location where the target data is located, the data processing capability of the site, and the network transmission capability between each two sites, wherein the target site is one or more of all the sites, the target data is the data that needs to be processed to execute the job, the data processing capability of the site is a comprehensive representation of various computing resources of the site, and the network transmission capability is the network bandwidth between the two sites; and an executing unit for executing the job execution plan so that the target site executes the job according to the job execution plan.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for processing geographically distributed jobs.

[0015] Applying the technical solution of the present application, the above-mentioned method for processing geographically distributed jobs first receives a job execution request, which is a request to execute a job and includes job information. Then, based at least on the job information, a job execution plan is generated, wherein the job execution plan is a plan that includes a target site for executing the job. The job execution plan is determined based on at least one of the site location where the target data is located, the site's data processing capacity, and the network transmission capacity between each two sites. The target site is one or more of all sites, the target data is the data to be processed for the job execution, the site's data processing capacity is the time it takes for the site to process the target data, and the network transmission capacity is the time it takes for the target data to be transmitted between the two sites. Finally, the job execution plan is executed so that the target site executes the job according to the job execution plan. This method coordinates computing jobs through a top-level manager context-aware scheduler to find the optimal execution plan for the job, so that the bottom-level sites process the job according to the plan, resulting in a more equitable workload distribution among the computing jobs. This solves the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for performing a processing method for geographically distributed jobs provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a geographically distributed scenario provided according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic flow chart of a method for processing geographically distributed jobs according to an embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram of job execution of a geographically distributed job provided according to an embodiment of the present application is shown;

[0021] Figure 5 A structural block diagram of a top-level manager and a bottom-level executor provided according to an embodiment of the present application is shown;

[0022] Figure 6 A schematic diagram of a geographically distributed scenario provided according to an embodiment of the present application is shown;

[0023] Figure 7A schematic diagram of multi-job scheduling of a geographically distributed job provided according to an embodiment of the present application is shown;

[0024] Figure 8 A schematic diagram showing one path in a processing path provided according to an embodiment of the present application is shown;

[0025] Figure 9 A schematic diagram of graphical modeling of multiple processing paths of multiple jobs of a geographically distributed job provided according to an embodiment of the present application is shown;

[0026] Figure 10 A topological diagram of three sites provided according to an embodiment of the present application is shown;

[0027] Figure 11 shows a reference topology diagram provided according to an embodiment of the present application;

[0028] Figure 12 A schematic diagram showing test results of a scenario in which data is evenly distributed among sites according to an embodiment of the present application is shown;

[0029] Figure 13 A schematic diagram showing test results for a scenario in which overall data provided by an embodiment of the present application is initially placed in site S4;

[0030] Figure 14 A schematic diagram showing test results for a scenario where data provided by an embodiment of the present application is initially placed at sites S1 and S3 is shown;

[0031] Figure 15 A structural block diagram of a geographically distributed job processing device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0036] H2F (Hierarchical Hadoop Framework): A hierarchical Hadoop framework for big data processing in geographically distributed environments.

[0037] Fog computing is an extension of cloud computing. The name "fog" (from "cloud") originates from the famous saying, "fog is the cloud closer to the ground." In this model, data, processing, and applications are concentrated in devices at the edge of the network, rather than being stored almost entirely in the cloud. Rather than relying on powerful servers, fog computing consists of less powerful, more distributed computers of various functions, embedded in factories, cars, appliances, streetlights, and other items in people's lives.

[0038] As introduced in the background, over the past two decades, we have witnessed an unprecedented global generation of data, which has created an ever-increasing demand for computing resources and spurred the design of computing paradigms and software tools to efficiently and quickly derive insights from this large amount of data. State-of-the-art parallel computing technologies, such as MapReduce, promise high performance in scenarios where the computing nodes involved are of equal size, clustered together via broadband networks, and the data is co-located with the cluster of nodes. Unfortunately, these technologies have proven ineffective in geographically distributed scenarios, i.e., computing environments where nodes and data are geographically distributed across multiple, often distant, data centers. Big data processing frameworks such as MapReduce and Hadoop are designed to efficiently analyze large datasets stored in a single data center. Apache Hadoop is the leading open source platform implementing the MapReduce parallel computing paradigm, leveraging the power provided by many computing nodes to accelerate data analysis. Hadoop only offers high computing performance for data stored in a single data center, implemented as a cluster of homogeneous computing resources connected by high-speed network links. Mining large amounts of data locally distributed across multiple, often distant, locations is a common requirement in these "geographic" scenarios. In this case, the computing resources are heterogeneous, with parts of the entire data set physically distributed across different locations and interconnected by network links that are much slower than the ones that interconnect the cluster nodes. None of the constraints under which Hadoop can guarantee performance improvements apply.

[0039] Therefore, to obtain effective and timely results, the analysis process of this geographically dispersed data must be thoroughly designed. Multinational retailers are accustomed to generating up to petabytes of data in a single day. The data generated by the company's outlets is transferred to geographically dispersed data centers and processed in a timely manner at these data centers. Similarly, because the data is locally distributed, the data processing strategies typically adopted in single-site scenarios are ineffective. Many IoT-based applications (such as climate monitoring and simulation) process large amounts of data sensed from multiple geographical locations. If the computing resources are located close to the sensors, as is now the case in many fog computing scenarios, then leveraging local computing power will be a wiser choice than moving all data from distributed locations to a central computing unit.

[0040] Faced with the aforementioned challenges, this application proposes a variant of the MapReduce paradigm that captures the constraints imposed in these scenarios (e.g., imbalances in node computing power and interconnect links) to implement intelligent job scheduling strategies. This application designs a hierarchical computing framework in which a context-aware scheduler coordinates computing jobs, leveraging the potential of the Hadoop framework within each participating data center. In this work, after demonstrating the characteristics of the developed framework, we intelligently segment the data so that the scheduler can produce a fairer workload distribution among the computing jobs.

[0041] In order to solve the problem of low data processing efficiency caused by uneven data distribution between different sites in a geographically distributed scenario, the embodiments of the present application provide a method for processing geographically distributed jobs, a processing system for geographically distributed jobs, a processing device for geographically distributed jobs, and a computer-readable storage medium.

[0042] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0043] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a mobile terminal for a method for processing geographically distributed jobs according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0044] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the device information display method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0045] In this embodiment, a method for processing geographically distributed jobs running on a mobile terminal, a computer terminal, or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] As mentioned above, in the MapReduce computing paradigm, a job is a computation request that can be submitted for processing. When a job is submitted, the scheduler splits it into many jobs, which are mapped to available compute nodes in the cluster. A typical metric for job processing performance is the throughput, or the time it takes to complete the job. In addition to the size of the data being processed, this throughput is influenced by (i) the job execution process designed by the scheduler (the job sequence) and (ii) the computing power provided by the cluster nodes hosting the job.

[0047] Let's take a specific scenario as an example: Company A operates in multiple countries around the world and has a point of presence (from now on, a site) in each country to collect data from the company's processes. Each site operates a local data center equipped with some computing resources. The computing resources at each site are interconnected via high-speed links (intra-site links) to form a powerful cluster. The sites are connected to each other via virtual private networks (VPNs) built on geographical network links (inter-site links). Furthermore, all resources (including computing and links) are virtualized to form a resource cloud that can be flexibly configured according to needs.

[0048] Assume that each site generates some raw data that needs to be mined, and that the amount of this data varies from site to site. Also assume that the computing power that each site can use for mining purposes varies from site to site. A snapshot of the situation just described is Figure 2 As shown in Figure 2. The company needs a way to quickly process all the distributed data using some parallel computing techniques that, at best, leverage the heterogeneous resources provided by its own computing infrastructure. Essentially, our proposal is based on the idea of ​​giving all sites a mining job proportional to their computing power; the more powerful a site is, the more data it needs to mine. Since the distribution of data does not follow the distribution of computing power (a large amount of data may happen to reside in low-power sites), data needs to be transferred from low-power sites (e.g., S1) to high-performance sites (e.g., S4).

[0049] Furthermore, the solution proposed in this application is application-agnostic, that is, it does not aim to design a solution that fits a specific application or application pattern. Instead, in the proposed model, the application is viewed as a black box that is able to consume some input data and generate some output data within a given time span, regardless of the algorithm that implements the data processing. The application is characterized by its "computation profile", which is the application's computational footprint calculated by observing the black box and analyzing the data input / output balance and processing time. Based on this profile and other parameters of the distributed context, the framework is able to recommend job schedules that optimize the performance of any application. Nevertheless, the framework does not prevent further performance enhancements from being achieved by improving the algorithms that implement the computations themselves.

[0050] Figure 3 FIG is a flow chart of a method for processing geographically distributed jobs according to an embodiment of the present application. Figure 3 As shown, the method includes the following steps:

[0051] Step S201: receiving a job execution request, wherein the job execution request is a request to execute a job, and the job execution request includes job information, wherein the job information includes at least one of the following: an algorithm for executing the job, input segments of the job, and a job description;

[0052] Specifically, the job execution request is issued by a requester user, and the execution subject of the method is a top-level manager.

[0053] Step S202: Generate a job execution plan based at least on the job information, wherein the job execution plan includes a plan representing a target site for executing the job, and is determined based on the site location where the target data is located, the data processing capability of the site, and the network transmission capability between each two sites. The target site is one or more of all the sites, the target data is data to be processed for executing the job, the data processing capability of the site is a comprehensive representation of various computing resources at the site, and the network transmission capability is the network bandwidth between the two sites.

[0054] Specifically, the above job execution plan can also be called a top-level job execution plan.

[0055] In an unbalanced computing environment, things become even more complicated. When submitting a job, the question is how to best schedule unbalanced resources across unevenly distributed data in a way that minimizes job completion time. We address the problem of context imbalance in parallel data computation by adopting a layered approach. In this approach, the top layer is responsible for running logic that is context-aware and orchestrates intelligent computational plans, while the bottom layer is responsible for executing the plans while leveraging well-known parallel computing techniques. Therefore, the top layer is where intelligent decisions must be made, such as which of many sparse data blocks a given computational resource should mine. Essentially, we believe that instead of having data computed by resources residing in the same data location, data should be moved to wherever the maximum possible computational power is available, provided that the benefits of utilizing the best computational resources outweigh the costs of moving the data closer to those resources. To support this level of intelligent decision-making, we designed and implemented a job execution plan.

[0056] In summary, H2F is based on the idea of ​​acquiring the most powerful computing resources to (a) attract as many data blocks as possible, (b) aggregate them, and (c) run a standard MapReduce program on the entire collected data. The proposed approach does not offer a new parallel computing paradigm for unbalanced computing environments. Instead, it aims to preserve and exploit the potential of current parallel computing technologies, even in complex computing environments, by creating two management levels that meet the needs of intelligent computing and fast computing, respectively.

[0057] like Figure 4 As shown, the computing sites are at the bottom of the hierarchy. Each site owns a certain amount of data and is capable of running ordinary Hadoop jobs. After receiving a sub-job request from the top level, the site executes the entire MapReduce execution flow on the local cluster and returns the results to the top level. The top-level manager owns the business logic of the system and is responsible for managing the unbalanced computing environment. After the top-level job is submitted, the business logic arranges the set of sub-jobs to be distributed throughout the distributed context, collects the sub-job results and packages the overall computation results. In the figure, the typical execution flow triggered by the top-level job submission is described by the numbered arrows. This particular case involves transferring data from one site to another (i.e., from S1 to S4) and running local MapReduce sub-jobs on two sites (S4 and S2). In the analyzed case, S3 did not participate in the computation. The following is Figure 4 Shown are step-by-step instructions for all actions the system must take to service the top-level job:

[0058] (1) The top-level manager receives a job execution request, which requires computing the data residing on S1, S2, and S4 respectively.

[0059] (2) The job execution plan is generated based on the following information: (a) the underlying state, such as the distribution of datasets between sites (the location of the target data at each site); (b) the current computing availability of the sites (the data processing capabilities of the sites); (c) the network topology and (d) the current capacity of its links (the network transmission capacity between each two sites).

[0060] (3) The top-level manager executes the job execution plan. According to the plan, it instructs S1 to transfer its data to S4.

[0061] (4) Actual data will be transferred from S1 to S4.

[0062] (5) According to the plan, the top-level manager sends a message to activate the sub-jobs running on the sites that currently store the data of interest. In particular, the top-level map jobs are triggered to run on S2 and S4 respectively (the top-level map jobs correspond to MapReduce sub-jobs).

[0063] (6) S2 and S4 execute local Hadoop sub-jobs on their respective datasets.

[0064] (7) S2 and S4 send the output of their local execution to the top-level manager.

[0065] (8) The processes run by the Top-Level Manager will be fed with partial results from the bottom-level detailed descriptions and perform global data calculations.

[0066] (9) The final output is returned to the requester user.

[0067] The entire process is transparent to the requester user, who only needs to provide the job and a pointer to the data to be processed. Figure 4 The design of the job execution process described in [1] takes into account the imbalance of the distributed computing infrastructure in terms of the computing power provided by the sites, the bandwidth of the network links between sites, and the data distribution. In this work, we assume that all sites are equipped with large-capacity storage devices to ensure comparable I / O performance.

[0068] The H2F logical architecture consists of several modules that handle Figure 4 Each site is essentially an independent system that runs its architecture instance in one of two different modes, depending on the role it plays. Of the participating sites, only one can activate orchestration mode, allowing it to act as a top-level manager; the top-level manager possesses the system's "intelligent" orchestration business logic and coordinates all system activities. The remaining sites must activate compute mode, allowing them to act as low-level executors—those who perform the work of data processing.

[0069] like Figure 5 As shown, to be a TopLevel Manager, you need to enable the following modules:

[0070] Master: It receives top-level job execution requests, extracts job information and passes it to the Orchestrator, which in turn uses it to build a job execution plan. The Master is responsible for executing the job execution plan and, after the Global Reducer has processed the job, delivers the final result to the requester.

[0071] Orchestrator: It builds job execution plans by combining information from submitted jobs and execution context (e.g., available computing power of sites and inter-site network capacity).

[0072] Global Reducer: It receives the output of sub-job computations from local sites and performs final computations.

[0073] To act as a bottom-level executor, you need to enable the following modules:

[0074] Node Coordinator: has all the information about the node (site) status.

[0075] File Manager: It handles chunk loading and chunk storage, and also keeps track of the file namespace.

[0076] Hadoop Handler: Basically, this module decouples the system from the underlying Hadoop-based computing platform. This allows the framework to be independent of the Hadoop flavor deployed in the local site.

[0077] Network Manager: It handles the communication between sites.

[0078] The job execution plan constructed by the Orchestrator includes instructions for data redistribution between sites and placement of child jobs. To construct the job execution plan, the orchestrator invokes a scheduling policy that predicts which execution path will ensure the highest job performance in terms of completion time.

[0079] The specific implementation steps of the above step S202 are as follows:

[0080] Step S2021: determining the target data according to target job information, wherein the target job information includes the job information or a plurality of sub-job information, wherein the sub-job information is obtained by splitting the job information;

[0081] Step S2022, obtaining the original site where the target data is located, the data processing capacity of each site, and the network transmission capacity between every two sites, where the original site is the site where the target data is stored;

[0082] Step S2023 : generating the job execution plan according to the target data, the original site where the target data is located, the data processing capability of each site, and the network transmission capability between every two sites.

[0083] Specifically, the job information can be split into multiple sub-job information or not. There is also a situation where the original site of the target data is the target site for processing the target job information, and splitting the job information into multiple sub-job information can increase the speed of job processing.

[0084] Wherein, generating the job execution plan further includes:

[0085] An optimization model is established based on the target data, wherein the optimization target of the optimization model is the time for processing the target data, and the variables of the optimization model include the original site where the target data is located and a preparatory site, the preparatory site is a site that processes the target job information, and the time for processing the target job information includes the sum of the data transmission time between the original site and the preparatory site, the job waiting time at the preparatory site, and the data processing time at the preparatory site;

[0086] Specifically, after a new job is submitted, the job scheduler will look for a site that is expected to maximize the performance of the job. If multiple jobs request the same site at the same time, a first-come, first-served approach is adopted. While the compute sites assigned to a job are intended to be dedicated to the job, this is not the case with network resources (site-to-site links), which are instead shared between jobs that need to move data between sites. Therefore, if the resource itself is accessed instantaneously by another job, the job may have to wait for the compute resource to become available. In addition, if it needs to move its target data across a network link shared with other jobs' data, it may experience further delays.

[0087] like Figure 6 and Figure 7 As shown in Figure 2, the target data can be logically and physically split into smaller data blocks of fixed size. After a job is submitted, its target data can be conveniently split and distributed to multiple sites, where the sub-jobs ultimately perform actual computations.

[0088] In order to properly service job execution requests, the scheduler must look up the resource usage plans of the jobs being served. In this example, we show three independent jobs being served based on their arrival times. Figure 7In

[15] , we described how the scheduler handles these three execution requests. Based on the data fragmentation principle, each job is divided into subjobs, which run on sites holding fragmented data. For example, Job1 is split into Job1.1, Job1.2, and Job1.3, running on sites S1, S2, and S3, respectively. In principle, subjobs must not begin execution until data is available on the sites. This is the case for Job1, whose start time is delayed (note the time lag between Job1's submission time and Job1.2's start time). Regarding Job3, its subjobs run immediately after Job3 is submitted because the job execution plan has not yet determined the data transfer. Notably, while Job1's subjobs may run immediately due to the availability of computing resources at the time of job submission, the start of Job3's subjobs is delayed until their computing resources become available (Job3.1 waits for Job1.2 to release S2, Job3.2 waits for Job2.2 to release S4, and Job3.3 waits for Job2.3 to release S5).

[0089] The specific steps for establishing the optimization model are as follows:

[0090] Step S301: Based on the target data, a plurality of nodes are formed. The optimization model further includes a plurality of nodes and a plurality of edges. The plurality of nodes of the optimization model include data computing nodes, data transmission nodes, and job waiting nodes. The data computing nodes represent data processing flow nodes of the preparatory site. The data transmission nodes represent data transmission flow nodes from the original site to the preparatory site. The job waiting nodes are job waiting flow nodes from the time the target data arrives at the preparatory site to the time it is processed. The preparatory site includes the original site.

[0091] Step S302: forming a plurality of edges representing the processing flow of the target data, and sequentially connecting the data transmission node, the job waiting node, and the data computing node using the edges to obtain a plurality of processing paths for the target data;

[0092] Step S303, determining a function for calculating the time taken for the processing path as the optimization target, and obtaining the optimization model;

[0093] Step S304: Determine, based on the optimization model, a feasible solution of the variable corresponding to the minimum optimization objective, and obtain the job execution plan.

[0094] Specifically, if Figure 8As shown, a processing path can include three types of nodes: the data transmission node, the job waiting node, and the data computing node; two types of nodes: the data transmission node and the data computing node; two types of nodes: the job waiting node and the data computing node; or only one type of node: the data computing node. The processing path is determined by the site location of the target data, the location of the target node, and the number of jobs processed by the target node.

[0095] To ensure the optimal execution path for the target job, the job scheduler leverages this model to automatically generate a finite number of potential target data processing paths and then searches for the path that minimizes the job's completion time. The target data processing path is modeled as a series of actions that drive job execution. Actions can be categorized into three types: site-to-site data transfer, on-site data refinement, and global reduction of all refined data. The target data processing path has many potential execution paths. We refer to these as "potential" execution paths because a job may follow many alternative paths during execution.

[0096] The basic idea is that each node represents an event that receives some input data, generates some output data (which may be of a different size than the input data), and consumes some time while doing so. Therefore, the execution time of the entire job represented in the graph can be estimated by calculating and collecting the time consumed by all nodes in the graph. The data computation element models data refinement through computation elements. It receives input data and outputs a data stream that is typically of a different size than the input data. The time consumed by this node depends on (a) the computational capabilities of the site it represents, (b) the type of computation performed by the job, and (c) the amount of data to be processed. The data transfer node models the transfer of data over a network link; therefore, the size of the output data does not differ from the size of the input data; the time consumed by this node is proportional to the bandwidth capacity of the link. The job waiting node models the delay incurred by a child job while waiting for available computational resources. The job waiting node also does not change the size of the data.

[0097] According to the optimization model, a feasible solution of the variable corresponding to the minimum of the optimization target is determined to obtain the job execution plan, including: calculating the value of the optimization target corresponding to each processing path; determining the processing path corresponding to the minimum value of the optimization target as the target path, determining the preparatory site corresponding to the target path as the target site, and determining the original site corresponding to the target path as the target original site, to obtain the job execution plan.

[0098] Wherein, when the original site and the preparatory site are different, calculating the value of the optimization target corresponding to each of the processing paths includes: obtaining the compression factor and throughput of the data computing node corresponding to the processing path to obtain the compression factor and the first throughput, obtaining the throughput of the data transmission node corresponding to the processing path to obtain the second throughput, and obtaining the waiting time of the target data on the job waiting node corresponding to the processing path, the compression factor being the ratio of the amount of data obtained by the preparatory site processing the target data to the amount of data of the target data, the first throughput being the amount of data processed by the preparatory site per unit time, and the second throughput being the amount of data transmitted from the original site per unit time. The site transmits the target data volume to the preliminary site, and the waiting time is the time the target data waits from arriving at the preliminary site to being processed; based on the target data and the compression factor, the data processing volume of the target site is determined to be the product of the target data and the compression factor; based on the data processing volume and the first throughput, the processing time is determined to be the ratio of the data processing volume to the first throughput; based on the target data and the second throughput, the transmission time is determined to be the ratio of the target data to the second throughput; based on the target data and the first throughput, the value of the optimization target corresponding to the processing path is determined to be the sum of the waiting time, the transmission time and the processing time.

[0099] like Figure 8 As shown, a node has two properties: a compression factor, βapp, which measures the data contraction / expansion introduced by the node; and throughput, defined as the amount of data per time unit that the node can process. The βapp value for data computation elements is equal to the ratio of output data to input data, while for data transmission nodes, βapp is equal to 1 because no data size modification occurs during data transmission. The throughput of data computation nodes depends on the computational power of the site and the type of computation performed by the job, while for data transmission nodes, the throughput corresponds to the link capacity. Finally, for job-waiting nodes, βapp is set to 1 (no computation is performed); and the concept of throughput does not exist for job-waiting nodes.

[0100] Typically, the execution time of a node is defined as the ratio of the input data size to the node throughput. Figure 8 The output data of the Nodej node in is obtained according to formula 1:

[0101] DataSize j，j+1 =DataSize j-1，j ×β j (Formula 1)

[0102] Among them, DataSize j,j+1Output data of Nodej node, DataSize j-1,j is the input data of Nodej node, β j is the compression factor of Nodej node

[0103] The execution time of a general node Nodej is obtained according to formula 2:

[0104]

[0105] Both βapp and throughput are application-specific, meaning they depend on the algorithm used to interpret the data for a particular job. Since these values ​​are not available when a job is submitted, we need to estimate them in advance. Specifically, before the job scheduler actually refines the job's execution plan, we run the job's algorithm on a small sample of the overall data that the job will actually refine. The results of this evaluation constitute what we call a job application profile.

[0106] Step S203: executing the job execution plan so that the target site executes the job according to the job execution plan.

[0107] Specifically, by formulating job execution plans at the top level and implementing them at the bottom level, accurate planning of data fragmentation and redistribution among the involved computing resources can improve the performance of MapReduce jobs in an unbalanced computing environment.

[0108] The specific implementation steps of step S203 are as follows:

[0109] Step S2031, when the target site is the original site of the target data, executing the job execution plan so that the target site processes the target data, and the original site is the site that stores the target data;

[0110] Step S2032 : If the target site is not the original site of the target data, execute the job execution plan to control the original site to send the target data to the target site so that the target site processes the target data.

[0111] Among them, if the target data is in the target site and the data processing capability of the target site is very strong, the target data can be directly processed in the target site without the need for data transmission.

[0112] In a case where the target site is not the original site of the target data, executing the job execution plan to control the original site to send the target data to the target site includes: controlling the original site to split the target data into multiple data blocks; and controlling the original site to send multiple data blocks to the target site.

[0113] Specifically, the execution path model is strongly influenced by the computational context. The number of potential paths generated for a given job depends strictly on: (a) the capacity of available compute nodes, (b) the number and capacity of network links, and (c) the data segmentation scheme. We split the dataset into equally sized chunks, facilitating the transfer of certain chunks from one site to another. On the one hand, data granularity provides the advantage of a more flexible data distribution scheme. As a counter-product, a large number of chunks yields a large number of potential execution paths. Essentially, the number of branches in a graph is the same as the number of executed MapReduce subjobs. Each branch originates from the root node (the initial node) and ends at a global reducer node. The execution time of a branch is the sum of the execution times of the nodes comprising the branch (except for the global reducer node, which is not included in this sum). We outline that execution along a branch is independent of each other; therefore, branches ultimately exhibit different execution times. The global reducer is activated only when all branches have output their results. Therefore, the slowest branch determines when global reducing is allowed to begin. Finally, the execution time of the global reducer is calculated by summing the output sizes received by the branches and dividing by the estimated throughput of the node.

[0114] In an embodiment where path modeling is optionally performed, we consider a scenario where Figure 6 The scenario described in Figure 7 The submission timing is shown. In addition, we assume that the data is ideally divided into 5GB blocks and each job needs to process 15GB of data. Job1 requests to process data residing in S1 (a 5GB block) and S3 (a 10GB block, which can be ideally split into two 5GB blocks); Job2 requests to process data residing in S1 (one block) and S5 (two blocks); Job3 requests to process data residing in S2, S4, and S5, each containing a 5GB block. Figure 9 In

[15] , we described a graph model representing two potential execution paths generated by the job scheduler for Job2 and Job3, respectively. For clarity, we did not describe the execution paths. The schedule for Job2 envisions three execution branches:

[0115] (1) The data block moves from S1 to S3 via links L111 and L113; here the child job Job2.1 suffers a delay because resource S3 is used by the child job Job1.3 (see Figure 7 ); once S3 is published, Job2.1 fetches and elaborates the data chunks; the output of the elaboration is transferred to S1 via links L113 and L111; here, a global reduction can be performed (note that the global reduction step will only begin after all the data elaborated by the child jobs has been collected in S1).

[0116] (2) The data block moves from S5 to S4, passing through links L225 and L224; here, Job2.2 immediately obtains computing resources (S4) and elaborates the data block; the refined output moves to S1 through links L224, L1122 and L111; here, global reduction can be performed.

[0117] (3) Job 2.3 accesses and elaborates the data block in S5; the refined output is transferred to S1 via the L225, L1122, and L111 links; here, a global reduction can be performed.

[0118] Similarly, the J3 plan envisions three branches. In this case, no data transfer is required in any of the branches. Notably, all child jobs experience delays because the requested computing resources are locked by other running child jobs. Note that the paths described are only potential execution paths for J2 and J3. The job scheduler is responsible for exploring all potential paths for the job and determining the path that provides the shortest execution time.

[0119] In one embodiment, the specific contents of controlling the original site to split the target data into multiple data blocks include: obtaining the computing power of the target site; obtaining the network transmission capacity between the original site and the target site; and determining the size of the data block based on the computing power of the original site and the network transmission capacity.

[0120] Specifically, the target data can be split into multiple data blocks of equal size, and then the data blocks can be moved between sites to transfer the data. Of course, data block size is not an independent variable; its impact must be considered together with other variables that characterize the distributed computing environment, namely the computing power of the target site and the network transmission capacity between the source site and the target site.

[0121] Moving data to a site is convenient if (a) the site provides a large amount of computing power (i.e., the computing power of the target site) and (b) the network connecting the target data's origin site and the target site has a large bandwidth (i.e., the network transmission capacity between the origin site and the target site). To measure the suitability of a site Site for this purpose, we calculate the computing power of site Si according to Formula 3:

[0122]

[0123] Among them, CPU rate is the proportion of the computing power of the target site in the computing power of all sites, CPU(Si) is the computing power of the target site, j=1NCPU(Sj) is the sum of the computing power of all sites.

[0124] The connection rate of site Si is defined as the arithmetic mean of all end-to-end nominal network bandwidths between the considered site and other sites in the distributed computing environment. Specifically, as shown in Formula 4:

[0125]

[0126] Among them, Bandwidth i,j is the nominal network capacity between nodes i and j, and N is the number of sites belonging to the distributed context. Connectivity rate is defined as the connectivity rate of a site, normalized to the overall connectivity of the distributed context. Therefore, the connectivity rate for site Si is obtained according to Equation 5:

[0127]

[0128] Among them, Connectivityrate(Si) is the connection rate of the site (that is, the proportion of the network transmission capacity of the site to the network transmission capacity of all sites), Connectivity(Si) is the network transmission capacity of the site, j=1NConnectivity(Sj) is the sum of the network transmission capacity of all sites.

[0129] Now, for each site, we calculate a score as a linear function of the above connection rate and CPUrate. The score function is given by:

[0130] Score(S i )=K×CPU rate (S i )+(1-K)×Connectivity rate (S i ) (Formula 6)

[0131] where K is an arbitrary constant in the range [0, 1]. We conducted several tests to optimally tune the value of K. In our scenario, good results were observed by setting K close to 0.5, a value that balances the contributions of network and computing power. However, finding the optimal value of K is beyond the scope of this work.

[0132] Finally, NominalBlockSize is obtained according to Formula 7:

[0133]

[0134] Where JobInputSize is the total amount of data to be processed by a given job. NominalBlockSize is the data chunk size found by the heuristic method. Of course, there's no guarantee that such a value is optimal (i.e., the value that maximizes job performance), but it's likely that the optimal value is close to the nominal chunk size. The "[0.5 × NominalBlockSize, MinChunkSize]" procedure is then run to find the optimal value close to the nominal chunk size.

[0135] Where MinChunkSize is the minimum size of a data chunk located at a site. Regarding the lower limit of the above range, we determined that 0.5 NominalBlockSize is a good compromise in seeking an optimal value close to NominalBlockSize while not causing excessive data fragmentation, which would ultimately harm overall performance. Finally, the data chunk size that resulted in the minimum job completion time was selected as the optimal candidate.

[0136] To better understand the mechanism, we provide the following example. Assume that the computing environment consists of three sites (S1, S2, and S3) that are interconnected via a star network topology, as shown in Figure 10 In this example, the total data to be processed is 10 GB, which initially resides on the following sites: S1←1GB, S2←3GB and S2←6GB.

[0137] The CPUrate, Connectivity, and Connectivityrate values ​​for site S1 are derived as follows:

[0138]

[0139]

[0140]

[0141] By setting K to 0.5, the score of site S1 is:

[0142] Score(S1)=0.5×CPU rate (S1)+0.5×Connectivity rate (S1)=0.33

[0143] Similarly, score (S2) and score (S3) are 0.271 and 0.43 respectively. The lowest score is site S2; according to our method, the selected NominalBlock size is:

[0144] NominalBlockSize=Score(S2)×JobInputSize

[0145] =0.271×10GB=271MB

[0146] In this case, the optimal size search process must look for the optimal size in the range [135MB, 1GB] because 1GB is the size of the smallest data block among the data blocks (located in S1).

[0147] In order to evaluate the effectiveness of the discussed heuristic approach, some experiments were performed. Figure 11 Starting with the reference scenario shown, we designed several configurations (intended to represent different specific computing scenarios) by adjusting the following parameters: the CPU power of each site, the capacity of the network links, and the initial distribution of data between the sites. These experiments were conducted by simulating the execution of a job that needs to process 10GB of data. Specifically, we considered a sample job with a βapp value estimated to be 0.5. The parameter under observation was, of course, the job completion time. The goal of the experiments was to demonstrate the ability of the job scheduler to derive an execution path that ensures the best job completion time by varying the data block size. Of course, this must be independent of the distributed computing scenario being considered.

[0148] We created five configurations, as shown in Table 1. Each configuration represents a specific distributed computing environment. Readers may notice that from Config1 to Config5, the computing scenarios become increasingly unbalanced.

[0149] Table 1. Test configuration

[0150]

[0151] In the test, for each configuration, the job scheduler receives context data. According to our mechanism, the optimal data chunk size is sought in the range [0.5×NominalBlockSize,MinChunkSize]. For the selected configurations, the calculation in Equation (3) gives the values ​​reported in Table 2.

[0152] Table 2. Nominal block sizes calculated for five configurations

[0153]

[0154] The program extracts a finite number of points that fall within the range, which form the points to be examined. Points are selected as follows: starting from the left edge of the range, the next point is obtained by stepping right by a fixed number of steps of 10% of the MinChunkSize. Of course, for a more refined search, smaller steps may be chosen: this will generate more points, which in turn means a higher burden on the job scheduler and no guarantee of consistent benefit.

[0155] In the first set of tests, the job input data was evenly distributed, so each site would host 1.7GB of data. The test results run on the five configurations are shown in Figure 1. Figure 12 As shown in Figure 2, the maximum makespan of the evaluation jobs is plotted as a function of the data block size. It is important to note that, for all configurations except Config1, the graph has a predominantly parabolic trend with a well-defined minimum, which is very close to the nominal block size.

[0156] In the second set of tests, the configuration remained almost unchanged; the only modification concerned the placement of the initial data. In particular, a single site was chosen to host the entire 10GB of data. For completeness, we repeated the tests, selecting a different site each time to host the data. It was observed that the initial placement of the data did not bias the experiment. Figure 13 In

[15] , we report the results for the case where the site chosen to host the data is S4. This time, the trend is more like a slope, but the minimum is still confined to the nominal block size.

[0157] In the last set of tests, we split the job's input data into two equal-sized chunks and distributed them across the two sites in the cluster. We repeated the experiment by considering each pair of sites in the scenario. It was observed that the choice of the two sites was uncorrelated. Figure 14 In , the results obtained by distributing the data uniformly over S1 and S3 are reported for each of the five configurations. The same considerations apply here as well.

[0158] In summary, it was shown that the optimal data chunk size depends on the connectivity of the computational context and the distribution of computing power. Based on this, the process of searching for the optimal data chunk size was embedded as a subroutine in the job scheduler to support the search for the best job execution path. Since the optimal data chunk size is always close to the nominal chunk size value, the exploration procedure focuses on a very limited boundary of this value (at most 3–4 points are examined).

[0159] The increasing amount of information generated daily by devices connected to the internet, combined with the massive amounts of data in traditional databases, has led to the definition of Big Data. MapReduce is perhaps the best-known paradigm, widely used in academic and commercial settings, for running parallel computations on Big Data. However, MapReduce cannot guarantee acceptable performance when it comes to computing data natively distributed across multiple heterogeneous computing resources.

[0160] This context-aware scheduler explores a large number of job schedules (execution paths) and proposes a suboptimal but very effective solution. The scheduler searches for the suboptimal data block size by exploring a subset of feasible data block size values. If on the one hand data fragmentation increases the complexity of the scheduling process, on the other hand fragmentation increases the chances of parallelizing the jobs for a given job. Looking closely at the results, we can notice that as the block size increases, the job completion time increases. We note that the data fragmentation and distribution schemes produced by the scheduler are always the result of a trade-off between the opportunity to utilize the most powerful computing resources and the constraints imposed by the worst performing network links.

[0161] The method for processing geographically distributed jobs in the present application first receives a job execution request, which is a request to execute a job and includes job information. Then, based at least on the job information, a job execution plan is generated, wherein the job execution plan is a plan that includes a target site for executing the job. The job execution plan is determined based on at least one of the site location where the target data is located, the site's data processing capacity, and the network transmission capacity between each two sites. The target site is one or more of all sites, the target data is the data to be processed for the job execution, the site's data processing capacity is the time it takes for the site to process the target data, and the network transmission capacity is the time it takes for the target data to be transmitted between two sites. Finally, the job execution plan is executed, causing the target site to execute the job according to the job execution plan. This method coordinates computing jobs through a top-level manager context-aware scheduler to find the optimal execution plan for the job, allowing bottom-level sites to process the job according to the plan, resulting in a more equitable workload distribution among computing jobs. This solves the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios.

[0162] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0163] The present application also provides a geographically distributed job processing system, such as Figure 5As shown, it includes: a top-level management structure for executing any one of the processing methods for geographically distributed jobs; a target site, which is used to execute the job according to the job execution plan, and the job execution plan is a plan including a target site that represents the execution of the job.

[0164] The aforementioned geographically distributed job processing system of the present application includes: a top-level management structure for executing any of the aforementioned geographically distributed job processing methods; and target sites for executing jobs according to a job execution plan, wherein the job execution plan is a plan that includes a representation of the target site for executing the job. The system coordinates computing jobs through a top-level manager's context-aware scheduler, finding the optimal execution plan for the jobs, enabling lower-level sites to process the jobs according to the plan and achieving a more equitable workload distribution among the computing jobs. This solves the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios.

[0165] The embodiments of the present application also provide a device for processing geographically distributed jobs. It should be noted that the device for processing geographically distributed jobs in the embodiments of the present application can be used to execute the processing method for geographically distributed jobs provided in the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation methods, and the details that have been explained will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0166] The following introduces the geographically distributed job processing device provided in the embodiment of the present application.

[0167] Figure 15 FIG is a schematic diagram of a processing device for geographically distributed jobs according to an embodiment of the present application. Figure 15As shown, the device includes a receiving unit 10, a generating unit 20, and an executing unit 30. The receiving unit 10 is used to receive a job execution request, which is a request to execute a job. The job execution request includes job information, and the job information includes at least one of the following: an algorithm for executing the job, an input segment of the job, and a job description; the generating unit 20 is used to generate a job execution plan based on at least the job information, wherein the job execution plan is a plan including a target site for executing the job, and the job execution plan is determined based on the site location where the target data is located, the data processing capability of the site, and the network transmission capability between each two sites, wherein the target site is one or more of all the sites, the target data is the data that needs to be processed to execute the job, the data processing capability of the site is a comprehensive representation of various computing resources of the site, and the network transmission capability is the network bandwidth between the two sites; the executing unit 30 is used to execute the job execution plan so that the target site executes the job according to the job execution plan.

[0168] As an optional solution, the generation unit includes a first determination module, a first acquisition module, and a first generation module. The first determination module is configured to determine the target data based on target job information, the target job information including the job information or multiple sub-job information, the sub-job information being obtained by splitting the job information. The first acquisition module is configured to acquire the original site where the target data is located, the data processing capacity of each site, and the network transmission capacity between each two sites, where the original site is the site where the target data is stored. The first generation module is configured to generate the job execution plan based on the target data, the original site where the target data is located, the data processing capacity of each site, and the network transmission capacity between each two sites. This can improve the speed of job processing.

[0169] Exemplarily, the first generation module includes an establishment module and a second determination module. The establishment module is used to establish an optimization model based on the target data. The optimization target of the optimization model is the time to process the target data. The variables of the optimization model include the original site and the preparatory site where the target data is located. The preparatory site is the site that processes the target job information. The time to process the target job information includes the sum of the data transmission time between the original site and the preparatory site, the job waiting time of the preparatory site, and the data processing time of the preparatory site. The second determination module is used to determine the feasible solution of the variables corresponding to the minimum optimization target based on the optimization model to obtain the job execution plan. By establishing the optimization model, a suboptimal but very effective solution is proposed, which makes it possible to find a better data processing path and improve job efficiency.

[0170] In an optional embodiment, the establishment module includes a first execution module, a second execution module and a third determination module, the first execution module is used to form a plurality of nodes according to the target data, the optimization model also includes a plurality of nodes and a plurality of edges, the plurality of nodes of the optimization model include data calculation nodes, data transmission nodes and job waiting nodes, the data calculation nodes represent the data processing flow nodes of the preparatory site, the data transmission nodes represent the data transmission flow nodes from the original site to the preparatory site, the job waiting nodes are the job waiting flow nodes from the time the target data arrives at the preparatory site to the time it is processed, wherein the preparatory site includes the original site; the second execution module is used to form a flow representing the target data The processing flow comprises multiple edges, and the edges are used to sequentially connect the data transmission nodes, the job waiting nodes, and the data computing nodes to obtain multiple processing paths for the target data; the third determination module is used to determine the function of the time taken to calculate the processing path as the optimization target, thereby obtaining the optimization model; and / or the second determination module further includes a calculation module and a fourth determination module, the calculation module is used to calculate the value of the optimization target corresponding to each processing path; the fourth determination module is used to determine the processing path corresponding to the minimum value of the optimization target as the target path, determine the preliminary site corresponding to the target path as the target site, and determine the original site corresponding to the target path as the target original site, thereby obtaining the job execution plan. The computing power and network transmission capacity between different sites can be balanced, making the best use of the most powerful site while reducing the cost of transferring data from one site to another.

[0171] In an optional scheme, the calculation module also includes a second acquisition module, a fifth determination module, a sixth determination module, a seventh determination module and an eighth determination module. The acquisition module is used to calculate the value of the optimization target corresponding to each processing path when the original site is different from the preparatory site, obtain the compression factor and throughput of the data calculation node corresponding to the processing path, obtain the compression factor and the first throughput, obtain the throughput of the data transmission node corresponding to the processing path, obtain the second throughput, and obtain the waiting time of the target data on the job waiting node corresponding to the processing path. The compression factor is the ratio of the amount of data obtained by the preparatory site processing the target data to the amount of data of the target data. The first throughput is the amount of data processed by the preparatory site per unit time. The second throughput is the unit The amount of data of the target data transmitted from the original site to the preliminary site within the bit time, the waiting time is the time the target data waits from arriving at the preliminary site to being processed; the fifth determination module is used to determine the data processing volume of the target site as the product of the target data and the compression factor based on the target data and the compression factor; the sixth determination module is used to determine the processing time as the ratio of the data processing volume to the first throughput based on the data processing volume and the first throughput; the seventh determination module is used to determine the transmission time as the ratio of the target data to the second throughput based on the target data and the second throughput; the eighth determination module is used to determine the value of the optimization target corresponding to the processing path as the sum of the waiting time, the transmission time and the processing time based on the target data and the first throughput. The time required for each process can be accurately calculated, thereby inferring the processing path with the shortest time.

[0172] In one optional solution, the execution unit includes a first execution subunit and a second execution subunit. The first execution subunit is configured to, when the target site is the original site of the target data, execute the job execution plan so that the target site processes the target data, and the original site is the site where the target data is stored. The second execution subunit is configured to, when the target site is not the original site of the target data, execute the job execution plan to control the original site to send the target data to the target site so that the target site processes the target data. Data can be moved to any location with the greatest possible computing power available, optimally scheduling unbalanced resources on unevenly distributed data in a manner that minimizes job completion time.

[0173] Exemplarily, the second execution subunit includes a first control module and a second control module. The first control module is configured to control the originating site to split the target data into multiple data blocks; the second control module is configured to control the originating site to send the multiple data blocks to the target site. This avoids the problem of uneven data size distributed throughout the computing context, which leads to low operation efficiency.

[0174] Exemplarily, the second control module includes a first acquisition submodule, a second acquisition submodule, and a determination submodule. The first acquisition submodule is configured to acquire the computing capacity of the target site; the second acquisition submodule is configured to acquire the network transmission capacity between the original site and the target site; and the determination submodule is configured to determine the size of the data block based on the computing capacity of the original site and the network transmission capacity. Searching for the optimal data block size can support searching for the optimal job execution path, reducing job execution time and improving job efficiency.

[0175] The above-mentioned geographically distributed job processing device of the present application includes: a receiving unit for receiving a job execution request, wherein the job execution request is a request to execute a job and includes job information; a generating unit for generating a job execution plan based at least on the job information, wherein the job execution plan is a plan including a target site for executing the job, and the job execution plan is determined based on at least one of the site location where the target data is located, the data processing capacity of the site, and the network transmission capacity between each two sites. The target site is one or more of all sites, the target data is the data to be processed for the job execution, the data processing capacity of the site is the time it takes for the site to process the target data, and the network transmission capacity is the time it takes for the target data to be transmitted between the two sites; and an executing unit for executing the job execution plan so that the target site executes the job according to the job execution plan. The device coordinates computing jobs through a top-level manager context-aware scheduler to find the optimal execution plan for the job, so that the bottom-level sites process the job according to the plan, resulting in a more equitable workload distribution among the computing jobs. This solves the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios.

[0176] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the aforementioned methods for processing geographically distributed jobs.

[0177] An embodiment of the present application also provides an electronic device, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a processing method for executing any one of the geographically distributed jobs.

[0178] The processing device for geographically distributed jobs includes a processor and memory. The aforementioned units are stored as program units in the memory, and the processor executes the program units stored in the memory to implement the corresponding functions. The aforementioned modules are all located in the same processor; alternatively, the aforementioned modules may be located in different processors in any combination.

[0179] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and kernel parameters can be adjusted to address the problem of uneven data distribution across different sites in geographically distributed scenarios, leading to low data processing efficiency.

[0180] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0181] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the processing method for the geographically distributed job.

[0182] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the method for processing geographically distributed jobs when running.

[0183] An embodiment of the present invention provides a device comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements at least the method for processing geographically distributed jobs. The device herein may be a server, a PC, a PAD, a mobile phone, or the like.

[0184] The present application also provides a computer program product which, when executed on a data processing device, is adapted to execute a program of the method steps of initiating at least the processing method of the geographically distributed job.

[0185] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0186] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0187] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0188] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0190] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0191] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0192] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0193] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0194] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0195] 1) The method for processing geographically distributed jobs described in the present application first receives a job execution request, which is a request to execute a job and includes job information. A job execution plan is then generated based at least on the job information, wherein the job execution plan is a plan that includes a representation of a target site for executing the job. The job execution plan is determined based on the site location where the target data is located, the site's data processing capabilities, and the network transmission capacity between each two sites. The target site is one or more of all sites, the target data is the data to be processed for the job execution, the site's data processing capabilities are a comprehensive representation of the site's various computing resources, and the network transmission capacity is the network bandwidth between the two sites. Finally, the job execution plan is executed so that the target site executes the job according to the job execution plan. This method coordinates computing jobs through a top-level manager context-aware scheduler to find the optimal execution plan for the job, allowing lower-level sites to process the job according to the plan, resulting in a fairer workload distribution among computing jobs. This solves the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios.

[0196] 2) The aforementioned geographically distributed job processing system of the present application includes: a top-level management structure for executing any of the aforementioned geographically distributed job processing methods; and target sites for executing jobs according to a job execution plan, wherein the job execution plan is a plan that includes a representation of the target site for executing the job. The system coordinates computing jobs through a top-level manager's context-aware scheduler, finding an optimal execution plan for the jobs, enabling lower-level sites to process the jobs according to the plan and achieving fairer workload distribution among computing jobs. This addresses the issue of low data processing efficiency caused by uneven data distribution across different sites in geographically distributed scenarios.

[0197] 3) The above-mentioned geographically distributed job processing device of the present application includes: a receiving unit for receiving a job execution request, wherein the job execution request is a request to execute a job and includes job information; a generating unit for generating a job execution plan based at least on the job information, wherein the job execution plan is a plan including a target site for executing the job, and the job execution plan is determined based on the site location where the target data is located, the data processing capacity of the site, and the network transmission capacity between each two sites. The target site is one or more of all sites, the target data is the data to be processed for the job execution, the data processing capacity of the site is a comprehensive representation of the various computing resources of the site, and the network transmission capacity is the network bandwidth between the two sites; and an executing unit for executing the job execution plan so that the target site executes the job according to the job execution plan. The device coordinates computing jobs through a top-level manager context-aware scheduler to find the optimal execution plan for the job, so that the bottom-level sites process the job according to the plan, resulting in a more equitable workload distribution among the computing jobs. This solves the problem of low data processing efficiency caused by uneven data distribution between different sites in geographically distributed scenarios.

[0198] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for processing geographically distributed jobs, characterized in that: include: Receive a job execution request, the job execution request being a request to execute a job, the job execution request including job information, the job information including at least one of the following: an algorithm for executing the job, an input segment of the job, and a job description; Generate a job execution plan based at least on the job information, wherein the job execution plan is a plan that includes a target site representing the job execution site, and the job execution plan is determined based on the site location where the target data is located, the data processing capability of the site, and the network transmission capability between each two sites, wherein the target site is one or more of all the sites, the target data is data to be processed to execute the job, the data processing capability of the site is a comprehensive representation of various computing resources at the site, and the network transmission capability is the network bandwidth between the two sites; executing the job execution plan so that the target site executes the job according to the job execution plan; Generating a job execution plan based at least on the job information includes: establishing an optimization model based at least on the target data, wherein the optimization objective of the optimization model is the time for processing the target data, and wherein variables of the optimization model include an original site where the target data is located and a preliminary site, wherein the preliminary site is a site for processing target job information; determining, based on the optimization model, a feasible solution for the variables that minimizes the optimization objective, thereby obtaining the job execution plan; In the case where the original site is different from the preparatory site, the value of the optimization target corresponding to each processing path is calculated, including: obtaining the compression factor and throughput of the data calculation node corresponding to the processing path to obtain the compression factor and the first throughput, obtaining the throughput of the data transmission node corresponding to the processing path to obtain the second throughput, and obtaining the waiting time of the target data on the job waiting node corresponding to the processing path, the compression factor being the ratio of the amount of data obtained by the preparatory site processing the target data to the amount of data of the target data, the first throughput being the amount of data processed by the preparatory site per unit time, and the second throughput being the amount of data transmitted from the original site to the job waiting node per unit time. The preparation site transmits the amount of target data, and the waiting time is the time the target data waits from the time it arrives at the preparation site to the time it is processed; based on the target data and the compression factor, the data processing volume of the target site is determined to be the product of the target data and the compression factor; based on the data processing volume and the first throughput, the processing time is determined to be the ratio of the data processing volume to the first throughput; based on the target data and the second throughput, the transmission time is determined to be the ratio of the target data to the second throughput; based on the target data and the first throughput, the value of the optimization target corresponding to the processing path is determined to be the sum of the waiting time, the transmission time and the processing time.

2. The processing method according to claim 1, characterized in that Generate a job execution plan based at least on the job information, including: Determining the target data according to target job information, where the target job information includes the job information or a plurality of sub-job information, where the sub-job information is obtained by splitting the job information; Obtaining an original site where the target data is located, data processing capabilities of each site, and network transmission capabilities between every two sites, where the original site is the site where the target data is stored; The job execution plan is generated according to the target data, the original site where the target data is located, the data processing capability of each site, and the network transmission capability between every two sites.

3. The processing method according to claim 2, characterized in that Generating the job execution plan according to the target data, the original site where the target data is located, the data processing capability of each site, and the network transmission capability between each two sites includes: The time for processing the target job information includes the sum of the data transmission time between the original site and the preparation site, the job waiting time of the preparation site, and the data processing time of the preparation site.

4. The processing method according to claim 3, characterized in that Establishing an optimization model based at least on the target data, including: According to the target data, a plurality of nodes are formed. The optimization model further includes a plurality of nodes and a plurality of edges. The plurality of nodes of the optimization model include data computing nodes, data transmission nodes, and job waiting nodes. The data computing nodes represent data processing flow nodes of the preparatory site. The data transmission nodes represent data transmission flow nodes from the original site to the preparatory site. The job waiting nodes are job waiting flow nodes from the time the target data arrives at the preparatory site to the time it is processed. The preparatory site includes the original site. forming a plurality of edges representing the processing flow of the target data, and sequentially connecting the data transmission node, the job waiting node, and the data computing node using the edges to obtain a plurality of processing paths for the target data; determining a function for calculating the time taken for the processing path as the optimization target, and obtaining the optimization model; and / or, Determining, based on the optimization model, a feasible solution for the variable corresponding to the minimum optimization objective, and obtaining the job execution plan, including: Calculating the value of the optimization target corresponding to each of the processing paths; Determine the processing path corresponding to the minimum value of the optimization target as the target path, determine the preparation site corresponding to the target path as the target site, and determine the original site corresponding to the target path as the target original site to obtain the job execution plan.

5. The processing method according to any one of claims 1 to 4, characterized in that Executing the job execution plan so that the target site executes the job according to the job execution plan includes: In a case where the target site is an original site of the target data, executing the job execution plan so that the target site processes the target data, and the original site is the site storing the target data; In a case where the target site is not the original site of the target data, the job execution plan is executed to control the original site to send the target data to the target site so that the target site processes the target data.

6. The processing method according to claim 5, characterized in that: In a case where the target site is not the original site of the target data, executing the job execution plan to control the original site to send the target data to the target site includes: Controlling the original site to split the target data into multiple data blocks; The original site is controlled to send a plurality of the data blocks to the target site.

7. The processing method according to claim 6, characterized in that Controlling the original site to split the target data into multiple data blocks includes: Obtaining the computing capacity of the target site; Acquiring the network transmission capacity between the original site and the target site; The size of the data block is determined according to the computing capability of the original site and the network transmission capability.

8. A geographically distributed job processing system, characterized in that: include: A top-level management structure for executing the method for processing geographically distributed jobs according to any one of claims 1 to 7; A target site is used to execute the job according to a job execution plan, wherein the job execution plan is a plan including a target site representing the job execution.

9. A geographically distributed job processing device, characterized in that: include: A receiving unit, configured to receive a job execution request, wherein the job execution request is a request to execute a job, and the job execution request includes job information, wherein the job information includes at least one of the following: an algorithm for executing the job, an input segment of the job, and a job description; a generating unit, configured to generate a job execution plan based at least on the job information, wherein the job execution plan is a plan including a target site representing a target site for executing the job, and the job execution plan is determined based on a site location where target data is located, a data processing capability of the site, and a network transmission capability between each two sites, wherein the target site is one or more of all the sites, the target data is data to be processed for executing the job, the data processing capability of the site is a comprehensive representation of various computing resources at the site, and the network transmission capability is the network bandwidth between the two sites; an execution unit, configured to execute the job execution plan so that the target site executes the job according to the job execution plan; The generating unit includes an establishing module and a second determining module, the establishing module being configured to establish an optimization model based at least on the target data, wherein the optimization objective of the optimization model is the time for processing the target data, and the variables of the optimization model include an original site where the target data is located and a preliminary site, wherein the preliminary site is a site for processing target job information; the second determining module being configured to determine, based on the optimization model, a feasible solution for the variables that minimizes the optimization objective, thereby obtaining the job execution plan; The second determination module also includes a calculation module, and the calculation module also includes a second acquisition module, a fifth determination module, a sixth determination module, a seventh determination module and an eighth determination module. The second acquisition module is used to calculate the value of the optimization target corresponding to each processing path when the original site is different from the preparatory site, obtain the compression factor and throughput of the data calculation node corresponding to the processing path, obtain the compression factor and the first throughput, obtain the throughput of the data transmission node corresponding to the processing path, obtain the second throughput, and obtain the waiting time of the target data on the job waiting node corresponding to the processing path. The compression factor is the ratio of the amount of data obtained by the preparatory site processing the target data to the amount of data of the target data. The first throughput is the amount of data of the target data processed by the preparatory site per unit time, and the second throughput is the unit throughput. the amount of data of the target data transmitted from the original site to the preliminary site within the bit time, and the waiting time is the time the target data waits from arriving at the preliminary site to being processed; the fifth determination module is used to determine, based on the target data and the compression factor, the data processing volume of the target site as the product of the target data and the compression factor; the sixth determination module is used to determine, based on the data processing volume and the first throughput, the processing time as the ratio of the data processing volume to the first throughput; the seventh determination module is used to determine, based on the target data and the second throughput, the transmission time as the ratio of the target data to the second throughput; the eighth determination module is used to determine, based on the target data and the first throughput, that the value of the optimization target corresponding to the processing path is the sum of the waiting time, the transmission time and the processing time.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for processing geographically distributed jobs according to any one of claims 1 to 7.

Images