Job execution method, apparatus and system, electronic device and program product

Abstract

The invention discloses a job execution method, device and system, electronic equipment and a program product, and relates to the field of distributed data processing.The method comprises the steps that a target job to be executed is determined, a front dependent job of the target job is determined, and a first job is obtained; when it is detected that execution of the first job is completed, the target job is fragmented, and a job fragment set of the target job is obtained; and processing the at least one job fragment through at least one computing node of the distributed system to complete the target job. Through the method and the device, the problem of relatively low batch job execution efficiency caused by executing batch jobs according to fixed moments in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of distributed data processing, and more specifically, to a method, device, system, electronic equipment and program product for executing a job. Background Art

[0002] In traditional scenarios, batch jobs are set to run at a fixed time, such as 1:00 a.m. every day. However, various factors may cause the job to run slowly. For example, there is a dependency between the previous and next jobs, and the next job starts late. For example, job A is the previous job and is set to run at 1:00, and job B is the next job and is set to run at 2:00. Job B can only be executed after job A is completed. If job A is executed at 1:30, but job B starts at 2:00, there is a half-hour difference between job A and job B, and half an hour of waiting time is wasted.

[0003] The existing technology may have the following problems: insufficient utilization of machine resources. For example, the machine has 10 CPU cores and 100GB of memory, but the job program only uses 1 core and 10GB of memory, and the remaining 90% of the machine resources are idle; insufficient utilization of machine resources. For example, the machine has 10 CPU cores and 100GB of memory, and the job program uses 10 cores and 100GB of memory, so all resources are fully occupied and execution takes a long time.

[0004] Currently, no effective solution has been proposed to solve the problem that batch jobs are executed at fixed times in related technologies, resulting in low batch job execution efficiency. Summary of the invention

[0005] The main purpose of the present application is to provide a job execution method, device, system, electronic device and program product to solve the problem in the related art that batch jobs are executed at fixed times, resulting in low batch job execution efficiency.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a job execution method is provided, which includes: determining a target job to be executed, and determining a predecessor dependent job of the target job to obtain a first job; when it is detected that the execution of the first job is completed, slicing the target job to obtain a job slice set of the target job, wherein the job slice set includes at least one job slice; processing the at least one job slice by at least one computing node of a distributed system to complete the target job.

[0007] Furthermore, determining a predecessor dependent job of the target job to obtain a first job includes: determining target application information, wherein the target application information represents application information of an application that executes the target job; determining dependent job information of the target job, wherein the dependent job information includes dependency information of at least one dependent job, and the dependency information includes at least: a dependent job and the number of executions of the dependent job; constructing a dependent data table based on the storage form of a data table according to the target application information and the dependent job information; and determining the first job according to the dependency data table.

[0008] Furthermore, the target job is sliced ​​to obtain a set of job slices of the target job, including: determining the data processing quantity of the target job; calculating the data processing quantity according to a preset algorithm to obtain the data processing quantity of each job slice; slicing the target job according to the data processing quantity of each job slice to obtain a set of job slices of the target job, wherein there is no dependency between each job slice in the set of job slices.

[0009] Furthermore, the target job is sliced ​​to obtain a job slice set of the target job, including: determining the data processing scope of the target job; determining each piece of data to be processed based on the data processing scope to obtain a set of data to be processed; splitting the set of data to be processed based on the regional information of each piece of data to be processed to obtain the data to be processed corresponding to each regional information; constructing the job slice corresponding to each regional information based on the data to be processed corresponding to each regional information to obtain the job slice set.

[0010] Furthermore, the at least one job slice is processed by at least one computing node of the distributed system to complete the target job, including: determining the amount of computing resources for the at least one job slice; determining the at least one computing node based on the amount of computing resources; allocating the at least one job slice to each computing node of the at least one computing node based on a load balancing algorithm to obtain a job slice corresponding to each computing node; and executing the job slice corresponding to each computing node by each computing node to complete the target job.

[0011] Furthermore, after processing the at least one job slice through at least one computing node of the distributed system, the method also includes: collecting the resource occupancy rate of each computing node in the at least one computing node; determining the target computing node based on the resource occupancy rate of each computing node, wherein the target computing node represents the computing node whose resource occupancy rate is greater than a preset value; obtaining a first computing node in the distributed system; and executing the job slice to be executed of the target computing node through the first computing node and the target computing node.

[0012] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a job execution system is provided, which includes: a database for storing a dependency data table, wherein the dependency data table includes at least dependent job information of the job and application information of the job; a controller for generating a scheduling instruction based on the dependency data table when receiving the execution result of a first job, and sending the scheduling instruction to at least one registered computing node; at least one computing node for receiving the scheduling instruction, and executing a second job according to the scheduling instruction to obtain the execution result of the second job; the at least one computing node is also used to send the execution result of the second job to the controller; a registration center for registering the at least one computing node to obtain the at least one registered computing node.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a job execution device is provided, which includes: a first determination unit, used to determine a target job to be executed, and determine a predecessor-dependent job of the target job to obtain a first job; a slicing unit, used to slice the target job when it is detected that the execution of the first job is completed, to obtain a job slice set of the target job, wherein the job slice set includes at least one job slice; a processing unit, used to process the at least one job slice through at least one computing node of a distributed system to complete the target job.

[0014] Furthermore, the first determination unit includes: a first determination subunit, used to determine target application information, wherein the target application information represents application information of an application that executes the target job; a second determination subunit, used to determine dependent job information of the target job, wherein the dependent job information includes dependency information of at least one dependent job, and the dependency information includes at least: dependent jobs and the number of executions of the dependent jobs; a first construction subunit, used to construct a dependent data table based on the storage form of a data table according to the target application information and the dependent job information; and a third determination subunit, used to determine the first job based on the dependency data table.

[0015] Furthermore, the slicing unit includes: a fourth determination subunit, used to determine the data processing quantity of the target job; a calculation subunit, used to calculate the data processing quantity according to a preset algorithm to obtain the data processing quantity of each job slice; a slicing subunit, used to slice the target job according to the data processing quantity of each job slice to obtain a job slice set of the target job, wherein there is no dependency between each job slice in the job slice set.

[0016] Furthermore, the sharding unit includes: a fifth determination subunit, used to determine the data processing scope of the target job; a sixth determination subunit, used to determine each piece of data to be processed based on the data processing scope, and obtain a set of data to be processed; a splitting subunit, used to split the set of data to be processed based on the regional information of each piece of data to be processed, and obtain the data to be processed corresponding to each regional information; a second construction subunit, used to construct the job slice corresponding to each regional information based on the data to be processed corresponding to each regional information, and obtain the job slice set.

[0017] Furthermore, the processing unit includes: a seventh determination subunit, used to determine the amount of computing resources for the at least one job slice; an eighth determination subunit, used to determine the at least one computing node based on the amount of computing resources; an allocation subunit, used to allocate the at least one job slice to each computing node of the at least one computing node based on a load balancing algorithm to obtain a job slice corresponding to each computing node; and an execution subunit, used to execute the job slice corresponding to each computing node through each computing node to complete the target job.

[0018] Furthermore, the device also includes: a collection unit, which is used to collect the resource occupancy rate of each computing node in the at least one computing node after processing the at least one job slice through at least one computing node of the distributed system; a second determination unit, which is used to determine the target computing node based on the resource occupancy rate of each computing node, wherein the target computing node represents a computing node whose resource occupancy rate is greater than a preset value; an acquisition unit, which is used to acquire a first computing node in the distributed system; and an execution unit, which is used to execute the job slice to be executed of the target computing node through the first computing node and the target computing node.

[0019] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements any one of the above-mentioned job execution methods, and which, when executed by a processor, implements the steps of the job execution method described in each embodiment of the present application.

[0020] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes stored computer instructions, wherein when the computer instructions are executed by a processor, any one of the above-mentioned job execution methods is implemented.

[0021] In order to achieve the above-mentioned purpose, according to one aspect of the present application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any one of the above-mentioned job execution methods.

[0022] In an embodiment of the present application, a first job is obtained by determining a target job to be executed and determining a predecessor dependent job of the target job; when it is detected that the first job is completed, the target job is sliced ​​to obtain a job slice set of the target job, wherein the job slice set includes at least one job slice; the at least one job slice is processed by at least one computing node of a distributed system to complete the target job, thereby achieving the purpose of quickly and accurately executing batch jobs, thereby realizing the technical effect of improving the execution efficiency of batch jobs, and further solving the technical problem of executing batch jobs at fixed times, resulting in low execution efficiency of batch jobs.

[0023] By clarifying the relationship between the target job and its pre-dependent jobs, it is possible to ensure that the target job is started after the dependent job (the first job) is completed, avoiding data errors or processing exceptions caused by the disorder of job execution order, and ensuring the logical coherence of the job process and the accuracy of data processing. At the same time, by slicing the data of the target job and assigning the job slices to multiple computing nodes for processing, the job can be decomposed into multiple subtasks, each of which can be executed in parallel on different computing nodes in the distributed system, and each computing node can focus on processing the specific job slice assigned to it, which can make more efficient use of computing resources, greatly shorten the overall execution time of the job, and not only improve the overall utilization of resources, but also improve the job processing speed, especially when processing large-scale data sets or high-concurrency tasks. The effect is particularly significant. Therefore, through dependency management and job slicing processing, combined with the parallel computing capabilities of the distributed system, the efficiency of job execution, resource utilization and system scalability can be significantly improved, while ensuring the accuracy of data processing and the automation of job management, achieving the effect of improving the performance of batch job execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 is a flowchart of a method for executing a job provided in Embodiment 1 of the present application;

[0026] Figure 2is a schematic diagram of an optional job execution method provided according to Embodiment 1 of the present application;

[0027] Figure 3 is a system structure diagram of a job execution system provided according to Embodiment 2 of the present application;

[0028] Figure 4 is a schematic diagram of a job execution device provided according to Embodiment 2 of the present application;

[0029] Figure 5 It is a schematic diagram of a job execution electronic device provided according to Embodiment 5 of the present application. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] It should be noted that the methods and devices determined by the processing methods, devices, storage media and electronic devices of the present application documents can be used in the financial technology field to execute batch jobs, thereby improving the efficiency of batch job execution, and can also be used in any field other than the financial technology field. The application fields of the methods and devices determined by the processing methods, devices, storage media and electronic devices of the present application documents are not limited.

[0032] It should be noted that the collected information (including but not limited to user device information, user personal information, collected data, used data, generated data, processed data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, collected information, used information, generated information, processed information, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0033] Example 1

[0034] According to an embodiment of the present application, a method embodiment of job execution is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a job execution method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (shown in the figure as 102a, 102 job execution, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (US job execution) port (which may be included as one of the ports of the job execution US bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0036] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the job execution method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned job execution method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0040] Under the above operating environment, this application provides Figure 2 The job execution method shown. Figure 2 It is a flowchart of the job execution method according to Example 1 of the present application.

[0041] The execution subject of the first embodiment of the present application is a job execution system. When processing job dependencies and resource allocation, the batch controller determines the execution mode of the job and the mode of allocating resources to different jobs according to the job definition and the information in the dependency table job_depend.

[0042] Optionally, a job execution system provided in Example 1 of the present application is characterized in that it includes: a database for storing a dependency data table, the dependency data table at least including the job's dependency job information and the job's application information; a controller for, upon receiving the execution result of a first job, generating a scheduling instruction based on the dependency data table, and sending the scheduling instruction to at least one registered computing node; at least one computing node for receiving the scheduling instruction, and executing a second job according to the scheduling instruction to obtain the execution result of the second job; the at least one computing node is also used to send the execution result of the second job to the controller; a registration center for registering at least one computing node to obtain at least one registered computing node.

[0043] In the first embodiment, the system structure diagram of the job execution system can be as follows: Figure 3As shown. The database is used to store dependency data tables. The dependency data table contains dependency information between jobs, that is, which jobs need to be completed before they can start executing (dependent job information), as well as the application information to which the job belongs, to identify the source and execution context of the job. For example, job B depends on job A, and the dependency data table in the database will record this relationship, as well as the application IDs to which jobs A and B belong respectively. The dependency data table includes but is not limited to the following fields: application ID, job ID, dependent application ID, dependent job ID, dependency type and version number, etc.

[0044] When the controller receives the execution result of the "first job" (i.e. the completed predecessor dependent job), it will query the dependency data table and analyze the job that should be started. Based on this information, the controller will generate a scheduling instruction to instruct the specific computing node when to start executing the "second job" that depends on the "first job". The controller sends the scheduling instruction to the registered computing nodes to ensure that the job can be executed according to the optimal path and resource allocation method.

[0045] At least one computing node is an entity that executes a specific job. They receive scheduling instructions from the controller and start executing the "second job" according to the instructions. After each computing node completes the execution, it sends the execution result of the "second job" back to the controller, forming a closed-loop feedback mechanism that enables the controller to continuously monitor the job status and make corresponding scheduling decisions.

[0046] The registration center is responsible for managing and registering computing nodes. Computing nodes need to register with the registration center before they can become valid computing nodes that receive scheduling instructions. The registration center maintains a list of registered computing nodes, which is visible to the controller so that the controller can know which computing nodes are available and perform resource scheduling and job allocation based on resource requirements and the current status of computing nodes.

[0047] Through the collaborative work of the above components, the entire job execution system can realize the automation of job execution, intelligent scheduling and efficient use of resources, thereby improving the efficiency and response speed of job execution and reducing waiting time and resource waste.

[0048] Step S201, determining a target job to be executed, and determining a preceding dependent job of the target job, to obtain a first job.

[0049] In this embodiment, in order to optimize the efficiency of job execution, the job execution system needs to identify the "target job" that needs to be executed. In the scenario of batch jobs, the target job can represent a job that has been pre-defined, such as job A that should be started at a preset time point (such as 1 am every day).

[0050] Next, the job execution system needs to analyze whether the target job has any pre-dependencies, that is, other jobs that need to be completed before the target job can start, that is, the "first job" mentioned above. For example, the target job B depends on the completion of job A before it can start execution. At this stage, the system will query the dependency table `job_depend` to find the pre-dependency job A related to the target job B.

[0051] By introducing dependent jobs, we can achieve seamless connection between jobs, reduce waiting time, improve resource utilization efficiency and the overall efficiency of job execution, and aim to solve efficiency problems caused by dependencies and improper resource utilization during job execution.

[0052] Step S202: when it is detected that the first job is completed, the target job is sliced ​​to obtain a job slice set of the target job, wherein the job slice set includes at least one job slice.

[0053] In the first embodiment, in order to solve the problem of insufficient resource utilization during job execution, after determining the predecessor dependent jobs, the job execution system will mark the predecessor dependent job that is completed first as the target job that can be started next, that is, the "first job" according to the priority and completion status of the dependency relationship. For example, if job A has been completed, then job B can be started immediately without waiting for a fixed time point. Job A is considered the "first job" at this time because it directly triggers the execution of job B.

[0054] Next, the job execution system will shard the data or tasks of the "target job" to generate a "job shard set", which contains at least one job shard. Sharding means breaking down a large job or task into multiple smaller task subsets, each of which is called a "job shard". For example, if the target job needs to process 1 million pieces of data, the controller may split it into 10 shards, each of which processes 100,000 pieces of data.

[0055] Through the above process, a single large job can be converted into multiple smaller job slices, which can be executed on multiple computing nodes at the same time. This parallel processing method improves the concurrency of job execution, fully utilizes computing resources, shortens the execution time of the overall job, and thus improves the efficiency of job execution.

[0056] Step S203: Process at least one job slice through at least one computing node of the distributed system to complete the target job.

[0057] In the first embodiment of the present invention, a distributed system refers to a system architecture in which multiple independent computing nodes (or computers) are connected through a network and work together to complete a common task. In this architecture, each computing node can independently process data or execute part of a task while maintaining communication with other nodes and sharing information. In the job execution system, at least one computing node is assigned to process job slices. These computing nodes are physical or virtual machines that actually execute jobs in the system, and they may be distributed in different geographical locations or on different servers.

[0058] After receiving the job slices assigned by the controller, the computing nodes start to independently execute the tasks in the slices. This parallel processing method fully utilizes the processing power of each computing node. Even if the target job is very large, it can be done through multiple computing nodes at the same time, greatly shortening the execution time.

[0059] The target job is considered complete when all assigned job slices are processed on their respective compute nodes. The compute nodes send the execution results of the job slices back to the controller, which can aggregate these results to form the final output of the target job.

[0060] By utilizing the parallel processing capabilities of distributed systems, the processing tasks of the target job can be decomposed onto multiple computing nodes, which can significantly improve the speed of job execution and the overall resource utilization of the system. This is one of the key technical solutions for optimizing the efficiency of batch job execution.

[0061] Optionally, in the job execution method provided in the first embodiment of the present application, determining a predecessor dependent job of a target job to obtain a first job includes: determining target application information, wherein the target application information represents application information of an application that executes the target job; determining dependent job information of the target job, wherein the dependent job information includes dependency information of at least one dependent job, and the dependency information includes at least: dependent jobs and the number of executions of the dependent jobs; constructing a dependent data table based on the storage form of a data table according to the target application information and the dependent job information; and determining the first job based on the dependency data table.

[0062] In the first embodiment, in order to ensure that the jobs can be executed in the correct order and with the best resource allocation, it is necessary to determine the specific application information of the target job. The target application information is metadata about the application, which may include the application ID, name, version, etc., to help the system identify which application's job needs to be processed. The target application information is generally stored in a database, for example, by querying the target application information table to obtain the target application information.

[0063] Next, the job execution system determines the target job's pre-dependent job information. The dependent job information includes at least the name of the dependent job and the number of executions, where the number of executions can also be called the job offset. The number of executions represents the number of times the dependent job needs to be successfully executed before the target job is started. For example, if the target job depends on "Job A" being successfully executed three times, then the target job can only start after the third successful execution of Job A.

[0064] The job execution system builds a dependency data table based on the target application information and dependent job information. This data table may be stored in a database and includes fields such as job ID, dependent job ID, and dependent job execution times to clearly express the dependency relationship between jobs.

[0065] Finally, the job execution system will determine the "first job", which is the previous dependent job of the target job, based on the information in the dependency data table. By analyzing the dependency data table, the controller can quickly identify the first job and start the target job in time to avoid waiting time and achieve seamless connection and efficient execution of jobs.

[0066] This series of operations ensures the consistency of job execution and efficient use of resources. Through dependency management, the entire job execution process is optimized and the execution efficiency of batch jobs is improved.

[0067] Optionally, in the job execution method provided in Example 1 of the present application, the target job is sliced ​​to obtain a set of job slices of the target job, including: determining the data processing quantity of the target job; calculating the data processing quantity according to a preset algorithm to obtain the data processing quantity of each job slice; slicing the target job according to the data processing quantity of each job slice to obtain a set of job slices of the target job, wherein there is no dependency between each job slice in the set of job slices.

[0068] In the first embodiment, in order to speed up the job processing speed, the job execution system needs to determine the total amount of data that the target job needs to process. For example, it is determined when the job is defined, or stored in the job definition table of the database. The amount of data to be processed may be the amount of data in the data table that needs to be processed, the number of files, or any other form of data.

[0069] The job execution system will use a preset algorithm to calculate the amount of data to be processed for each job slice after the target job is sliced. The preset algorithm may take into account factors such as the number of computing nodes, resource allocation strategy, load balancing of data processing, and possible hardware limitations. The goal of the preset algorithm is to distribute the total amount of data evenly or according to the strategy to ensure that the amount of data processed by the job slice on each computing node is appropriate to avoid overload or waste of resources. For example, if the target job needs to process 1 million data, in order to ensure the balance of parallel processing, the job execution system can divide it into 10 job slices, and each slice will process 100,000 data.

[0070] Finally, according to the calculated data processing quantity of each job slice, the target job is divided into multiple independent job slices. Each job slice contains a specific part of data or tasks, which can be processed independently and have no dependencies on each other, which means that the execution of a slice does not need to wait for the completion of other slices, so they can be executed in parallel. After the slicing operation, a "job slice collection" is generated, which contains all independent job slices. Each job slice is part of the target job and will be assigned to the computing nodes in the distributed system for processing. The generation of a job slice collection is the key to achieving parallel processing, which allows the system to accelerate the completion of jobs through multi-node parallel execution.

[0071] Through the above steps, the job execution system can effectively decompose resource-intensive or large-data-volume target jobs into multiple parallel job slices, thereby fully utilizing the computing power of the distributed system and improving the efficiency and speed of job execution.

[0072] Optionally, in the job execution method provided in Example 1 of the present application, the target job is sliced ​​to obtain a set of job slices of the target job, including: determining the data processing scope of the target job; determining each piece of data to be processed based on the data processing scope to obtain a set of data to be processed; splitting the set of data to be processed based on the regional information of each piece of data to be processed to obtain the data to be processed corresponding to each regional information; constructing the job slice corresponding to each regional information based on the data to be processed corresponding to each regional information to obtain a set of job slices.

[0073] In the first embodiment of the present invention, in order to further optimize the distribution and efficiency of job processing, the job execution system needs to clarify the data range that the target job needs to process. For example, the specific starting and ending positions, time windows, or specific data sets in the data table. The determination of the data processing scope is the starting point of job execution, which is based on the business requirements of the job and the actual situation of the data. After clarifying the data processing scope, the job execution system will list all data items that need to be processed by the target job to form a "data set to be processed". This set contains a list of all the data to be processed by the target job, and each piece of data is an element in the set.

[0074] Next, the job execution system splits the data set to be processed according to the regional information of each piece of data (for example, the geographical location of the data source). Regional information is used to distinguish the physical or logical location of the data, which ensures that data processing can be performed on geographically close or logically related computing nodes, thereby reducing network latency and data transmission costs. After splitting, a subset of the data to be processed for each regional information will be obtained. When constructing job shards, factors such as the load of the computing nodes and resource allocation strategies may be further considered to achieve optimal resource utilization and processing speed.

[0075] Finally, the code execution system will generate a set of job slices corresponding to all regional information, namely the "job slice set". Each job slice in this set is an independent data processing unit that can be executed in parallel, and they will be assigned to the appropriate computing nodes in the distributed system for processing.

[0076] By sharding jobs based on region information, the job execution system can achieve more detailed and efficient resource allocation, especially for jobs involving large amounts of data or requiring high geographic relevance. This sharding strategy can reduce data transmission time, increase processing speed, and ensure the logical coherence and consistency of data processing.

[0077] Optionally, in the job execution method provided in Example 1 of the present application, at least one job slice is processed by at least one computing node of the distributed system to complete the target job, including: determining the amount of computing resources for at least one job slice; determining at least one computing node based on the amount of computing resources; allocating at least one job slice to each computing node of at least one computing node based on a load balancing algorithm to obtain a job slice corresponding to each computing node; and executing the job slice corresponding to each computing node through each computing node to complete the target job.

[0078] In the first embodiment of the present invention, the job execution system needs to evaluate the computing resources required for the execution of each job slice, such as CPU, memory, disk I / O, etc. This is usually estimated based on the job type, historical execution data, and the size of the data in the job slice. The determination of the amount of computing resources is crucial for subsequent resource allocation and node selection. Once the computing resource requirements of each job slice are evaluated, the system will determine the appropriate computing nodes based on these requirements. Computing nodes are physical or virtual servers in a distributed system that have different computing resources. The job execution system will select nodes that meet the job resource requirements, and may select multiple nodes to process different job slices to achieve parallel execution.

[0079] Next, the job execution system uses a load balancing algorithm to distribute job slices to the compute nodes. The load balancing algorithm ensures that the load (i.e., the amount of tasks) of all nodes is as even as possible, avoiding the situation where some nodes are overloaded while other nodes are idle. By properly distributing job slices, the load balancing algorithm can maximize the efficiency of system resource utilization and reduce job execution time. After the load balancing algorithm is executed, an allocation plan will be obtained, that is, a list of job slices that each compute node will process. This allocation plan ensures that each node has enough workload while avoiding resource waste or bottlenecks.

[0080] Finally, the computing nodes start to execute data processing tasks according to the assigned job slices. Each node processes its assigned job slices independently, achieving parallel execution and significantly speeding up the processing speed of the target job. When all computing nodes complete the processing of their corresponding job slices, the target job is considered completed.

[0081] Through resource evaluation, node selection and load balancing, the high efficiency, parallelism and rational use of resources of job execution in the distributed system are ensured. By distributing job slices in parallel to multiple computing nodes, the system can significantly shorten the total execution time of the job and improve the overall processing capacity.

[0082] Optionally, in the job execution method provided in Example 1 of the present application, after processing at least one job slice through at least one computing node of the distributed system, the method also includes: collecting the resource occupancy rate of each computing node in the at least one computing node; determining the target computing node based on the resource occupancy rate of each computing node, wherein the target computing node represents a computing node whose resource occupancy rate is greater than a preset value; obtaining a first computing node in the distributed system; and executing the job slice to be executed of the target computing node through the first computing node and the target computing node.

[0083] In the first embodiment, in order to monitor and optimize the resource usage of computing nodes in the distributed system and ensure that the job slices can be efficiently executed on nodes with sufficient resources, the job execution system regularly collects the resource usage of all computing nodes in the distributed system, including CPU usage, memory usage, disk I / O and other indicators. The resource usage can be collected through a preset monitoring tool or reported by the computing node itself, and these data are used for subsequent resource allocation decisions.

[0084] The job execution system analyzes the resource utilization of each computing node and identifies computing nodes whose resource utilization is higher than the preset threshold. These nodes are called "target computing nodes". The preset threshold is a set value. When the resource utilization of a computing node reaches or exceeds this value, it indicates that the node may be overloaded and measures need to be taken to distribute its workload.

[0085] The job execution system will select computing nodes with low resource usage or that meet other preset conditions from the distributed system. These nodes are called "first computing nodes". The purpose of selecting "first computing nodes" is to find nodes with sufficient resources to execute job slices, ensuring that the job slices can be completed quickly and smoothly.

[0086] Finally, the job execution system will transfer the job slices to be executed on the "target computing node" to the "first computing node" for execution, or start a new instance on the "first computing node" to execute the job slices in parallel. This resource scheduling mechanism helps balance the load in the system, avoid overloading the computing nodes, and utilize idle computing resources to improve job processing speed and efficiency.

[0087] Through the above steps, the distributed job execution system can monitor the resource usage of computing nodes in real time and dynamically adjust the execution location of job slices as needed to ensure efficient resource utilization while avoiding the decline in execution efficiency due to resource overload. This mechanism is particularly important for processing large-scale data or high-concurrency job scenarios, and can significantly improve the overall performance of the job execution system and the reliability of job execution.

[0088] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0089] In summary, the job execution method provided in the embodiment of the present application obtains a first job by determining a target job to be executed and determining a predecessor dependent job of the target job; when it is detected that the execution of the first job is completed, the target job is sliced ​​to obtain a job slice set of the target job, wherein the job slice set includes at least one job slice; at least one job slice is processed by at least one computing node of a distributed system to complete the target job, thereby solving the problem in the related art of executing batch jobs at fixed times, resulting in low execution efficiency of batch jobs.

[0090] By clarifying the relationship between the target job and its pre-dependent jobs, it is possible to ensure that the target job is started after the dependent job (the first job) is completed, avoiding data errors or processing exceptions caused by the disorder of job execution order, and ensuring the logical coherence of the job process and the accuracy of data processing. At the same time, by slicing the data of the target job and assigning the job slices to multiple computing nodes for processing, the job can be decomposed into multiple subtasks, each of which can be executed in parallel on different computing nodes in the distributed system, and each computing node can focus on processing the specific job slice assigned to it, which can make more efficient use of computing resources, greatly shorten the overall execution time of the job, and not only improve the overall utilization of resources, but also improve the job processing speed, especially when processing large-scale data sets or high-concurrency tasks. The effect is particularly significant. Therefore, through dependency management and job slicing processing, combined with the parallel computing capabilities of the distributed system, the efficiency of job execution, resource utilization and system scalability can be significantly improved, while ensuring the accuracy of data processing and the automation of job management, achieving the effect of improving the performance of batch job execution.

[0091] Example 2

[0092] The embodiment of the present application also provides a job execution device. It should be noted that the job execution device of the embodiment of the present application can be used to execute the job execution method provided by the embodiment of the present application. The job execution device provided by the embodiment of the present application is introduced below.

[0093] According to an embodiment of the present application, a device for implementing the above-mentioned job execution method is also provided, such as Figure 4 As shown, the device comprises:

[0094] Specifically, the first determining unit 401 is used to determine a target job to be executed, and determine a preceding dependent job of the target job, to obtain a first job.

[0095] The slicing unit 402 is used to slice the target job to obtain a set of job slices of the target job when it is detected that the first job is completed, wherein the set of job slices includes at least one job slice.

[0096] The processing unit 403 is used to process at least one job slice through at least one computing node of the distributed system to complete the target job.

[0097] The job execution device provided in the embodiment of the present application determines the target job to be executed through the first determination unit 401, and determines the predecessor dependent job of the target job to obtain the first job; the slicing unit 402 slices the target job when detecting that the execution of the first job is completed, to obtain a job slice set of the target job, wherein the job slice set includes at least one job slice; the processing unit 403 processes at least one job slice through at least one computing node of the distributed system to complete the target job, thereby solving the problem in the related art that batch jobs are executed at fixed times, resulting in low execution efficiency of batch jobs.

[0098] By clarifying the relationship between the target job and its pre-dependent jobs, it is possible to ensure that the target job is started after the dependent job (the first job) is completed, avoiding data errors or processing exceptions caused by the disorder of job execution order, and ensuring the logical coherence of the job process and the accuracy of data processing. At the same time, by slicing the data of the target job and assigning the job slices to multiple computing nodes for processing, the job can be decomposed into multiple subtasks, each of which can be executed in parallel on different computing nodes in the distributed system, and each computing node can focus on processing the specific job slice assigned to it, which can make more efficient use of computing resources, greatly shorten the overall execution time of the job, and not only improve the overall utilization of resources, but also improve the job processing speed, especially when processing large-scale data sets or high-concurrency tasks. The effect is particularly significant. Therefore, through dependency management and job slicing processing, combined with the parallel computing capabilities of the distributed system, the efficiency of job execution, resource utilization and system scalability can be significantly improved, while ensuring the accuracy of data processing and the automation of job management, achieving the effect of improving the performance of batch job execution.

[0099] Optionally, in the job execution device provided in Example 2 of the present application, the above-mentioned first determination unit 401 includes: a first determination sub-unit, used to determine target application information, wherein the target application information represents application information of an application that executes the target job; a second determination sub-unit, used to determine dependent job information of the target job, wherein the dependent job information includes dependency information of at least one dependent job, and the dependency information includes at least: dependent jobs, and the number of executions of the dependent jobs; a first construction sub-unit, used to construct a dependency data table based on the storage form of the data table, according to the target application information and the dependent job information; and a third determination sub-unit, used to determine the first job based on the dependency data table.

[0100] Optionally, in the job execution device provided in Example 2 of the present application, the above-mentioned slicing unit 402 includes: a fourth determination sub-unit, used to determine the data processing quantity of the target job; a calculation sub-unit, used to calculate the data processing quantity according to a preset algorithm to obtain the data processing quantity of each job slice; a slicing sub-unit, used to slice the target job according to the data processing quantity of each job slice to obtain a job slice set of the target job, wherein there is no dependency between each job slice in the job slice set.

[0101] Optionally, in the job execution device provided in Example 2 of the present application, the above-mentioned slicing unit 402 includes: a fifth determination sub-unit, used to determine the data processing scope of the target job; a sixth determination sub-unit, used to determine each to-be-processed data based on the data processing scope, and obtain a set of to-be-processed data; a splitting sub-unit, used to split the set of to-be-processed data based on the regional information of each to-be-processed data, and obtain the to-be-processed data corresponding to each regional information; a second construction sub-unit, used to construct the job slice corresponding to each regional information based on the to-be-processed data corresponding to each regional information, and obtain a set of job slices.

[0102] Optionally, in the job execution device provided in Example 2 of the present application, the above-mentioned processing unit 403 includes: a seventh determination subunit, used to determine the amount of computing resources by determining at least one job slice; an eighth determination subunit, used to determine at least one computing node based on the amount of computing resources; an allocation subunit, used to allocate at least one job slice to each computing node of at least one computing node based on a load balancing algorithm, to obtain a job slice corresponding to each computing node; an execution subunit, used to execute the job slice corresponding to each computing node through each computing node to complete the target job.

[0103] Optionally, in the job execution device provided in Example 2 of the present application, the above-mentioned device also includes: a collection unit, which is used to collect the resource occupancy rate of each computing node in at least one computing node after processing at least one job slice through at least one computing node of the distributed system; a second determination unit, which is used to determine the target computing node based on the resource occupancy rate of each computing node, wherein the target computing node represents a computing node whose resource occupancy rate is greater than a preset value; an acquisition unit, which is used to acquire a first computing node in the distributed system; and an execution unit, which is used to execute the job slice to be executed of the target computing node through the first computing node and the target computing node.

[0104] It should be noted that the first determination unit 401, the slicing unit 402 and the processing unit 403 correspond to steps S201 to S203 in Example 1, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102 job execution, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0105] Example 3

[0106] An embodiment of the present application may provide an electronic device, Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (only one is shown) processor 502, memory 504, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0107] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the target job to be executed, and determine the predecessor dependent job of the target job to obtain a first job; when it is detected that the first job is completed, slice the target job to obtain a job slice set of the target job, wherein the job slice set includes at least one job slice; and process the at least one job slice through at least one computing node of the distributed system to complete the target job.

[0109] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the predecessor dependent job of the target job to obtain the first job, including: determine the target application information, wherein the target application information represents the application information of the application that executes the target job; determine the dependent job information of the target job, wherein the dependent job information includes dependency information of at least one dependent job, and the dependency information includes at least: the dependent job and the number of executions of the dependent job; based on the storage form of the data table, construct a dependent data table according to the target application information and the dependent job information; determine the first job according to the dependent data table.

[0110] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: slice the target job to obtain a set of job slices of the target job, including: determining the data processing quantity of the target job; calculating the data processing quantity according to a preset algorithm to obtain the data processing quantity of each job slice; slice the target job according to the data processing quantity of each job slice to obtain a set of job slices of the target job, wherein there is no dependency between each job slice in the set of job slices.

[0111] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: slice the target job to obtain a set of job slices of the target job, including: determining the data processing scope of the target job; determining each piece of data to be processed based on the data processing scope to obtain a set of data to be processed; splitting the set of data to be processed based on the regional information of each piece of data to be processed to obtain the data to be processed corresponding to each regional information; constructing the job slice corresponding to each regional information based on the data to be processed corresponding to each regional information to obtain a set of job slices.

[0112] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: processing at least one job slice through at least one computing node of the distributed system to complete the target job, including: determining the amount of computing resources for at least one job slice; determining at least one computing node based on the amount of computing resources; allocating at least one job slice to each computing node of at least one computing node based on the load balancing algorithm to obtain the job slice corresponding to each computing node; executing the job slice corresponding to each computing node through each computing node to complete the target job.

[0113] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: After processing at least one job slice through at least one computing node of the distributed system, the method also includes: collecting the resource occupancy rate of each computing node in the at least one computing node; determining the target computing node based on the resource occupancy rate of each computing node, wherein the target computing node represents a computing node whose resource occupancy rate is greater than a preset value; obtaining a first computing node in the distributed system; and executing the job slice to be executed of the target computing node through the first computing node and the target computing node.

[0114] According to an embodiment of the present application, a first job is obtained by determining a target job to be executed and determining a predecessor dependent job of the target job; when it is detected that the first job has been completed, the target job is sliced ​​to obtain a set of job slices of the target job, wherein the set of job slices includes at least one job slice; at least one job slice is processed by at least one computing node of a distributed system to complete the target job, thereby solving the problem in related technologies of executing batch jobs at fixed times, resulting in low execution efficiency of batch jobs.

[0115] By clarifying the relationship between the target job and its pre-dependent jobs, it is possible to ensure that the target job is started after the dependent job (the first job) is completed, avoiding data errors or processing exceptions caused by the disorder of job execution order, and ensuring the logical coherence of the job process and the accuracy of data processing. At the same time, by slicing the data of the target job and assigning the job slices to multiple computing nodes for processing, the job can be decomposed into multiple subtasks, each of which can be executed in parallel on different computing nodes in the distributed system, and each computing node can focus on processing the specific job slice assigned to it, which can make more efficient use of computing resources, greatly shorten the overall execution time of the job, and not only improve the overall utilization of resources, but also improve the job processing speed, especially when processing large-scale data sets or high-concurrency tasks. The effect is particularly significant. Therefore, through dependency management and job slicing processing, combined with the parallel computing capabilities of the distributed system, the efficiency of job execution, resource utilization and system scalability can be significantly improved, while ensuring the accuracy of data processing and the automation of job management, achieving the effect of improving the performance of batch job execution.

[0116] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), or a PAD. Figure 5 The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations shown.

[0117] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0118] Example 4

[0119] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the job execution method provided in the first embodiment.

[0120] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0121] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of the job execution method.

[0122] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0123] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0128] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for executing a job, characterized in that: include: Determine a target job to be executed, and determine a pre-dependent job of the target job to obtain a first job; When it is detected that the first job is completed, the target job is sliced ​​to obtain a set of job slices of the target job, wherein the set of job slices includes at least one job slice; The at least one job slice is processed by at least one computing node of the distributed system to complete the target job.

2. The method according to claim 1, characterized in that Determining a pre-dependent job of the target job to obtain a first job includes: Determine target application information, wherein the target application information represents application information of an application that executes the target job; Determine dependent job information of the target job, wherein the dependent job information includes dependency information of at least one dependent job, and the dependency information at least includes: a dependent job and the number of executions of the dependent job; Based on the storage form of the data table, construct a dependent data table according to the target application information and the dependent job information; The first operation is determined according to the dependency data table.

3. The method according to claim 1, characterized in that The target job is sliced ​​to obtain a set of job slices of the target job, including: Determining the data processing quantity of the target job; Calculate the data processing quantity according to a preset algorithm to obtain the data processing quantity of each job slice; The target job is sliced ​​according to the data processing quantity of each job slice to obtain a job slice set of the target job, wherein no dependency exists between each job slice in the job slice set.

4. The method according to claim 1, characterized in that: The target job is sliced ​​to obtain a set of job slices of the target job, including: Determining a data processing scope of the target job; Determine each piece of data to be processed according to the data processing scope to obtain a set of data to be processed; Splitting the set of data to be processed according to the regional information of each piece of data to be processed to obtain the data to be processed corresponding to each piece of regional information; The job slices corresponding to each region information are constructed according to the to-be-processed data corresponding to each region information to obtain the job slice set.

5. The method according to claim 1, characterized in that Processing the at least one job slice by at least one computing node of the distributed system to complete the target job includes: Determine the at least one job slice to determine the amount of computing resources; Determining the at least one computing node according to the amount of computing resources; Allocating the at least one job slice to each computing node of the at least one computing node according to a load balancing algorithm to obtain a job slice corresponding to each computing node; The target job is completed by executing the job slice corresponding to each computing node through each computing node.

6. The method according to claim 1, characterized in that After processing the at least one job slice by at least one computing node of the distributed system, the method further includes: Collecting resource occupancy of each computing node in the at least one computing node; Determine a target computing node according to the resource occupancy rate of each computing node, wherein the target computing node represents a computing node whose resource occupancy rate is greater than a preset value; Acquire a first computing node in the distributed system; The job slice to be executed of the target computing node is executed through the first computing node and the target computing node.

7. A job execution system, characterized in that: include: A database, used to store a dependency data table, wherein the dependency data table includes at least dependent job information of a job and application information of the job; A controller, configured to generate a scheduling instruction according to the dependency data table upon receiving the execution result of the first job, and send the scheduling instruction to at least one registered computing node; at least one computing node, configured to receive the scheduling instruction, execute the second job according to the scheduling instruction, and obtain an execution result of the second job; The at least one computing node is further used to send the execution result of the second job to the controller; The registration center is used to register the at least one computing node and obtain the at least one registered computing node.

8. A work execution device, characterized in that: include: A first determining unit, configured to determine a target job to be executed, and determine a pre-dependent job of the target job, to obtain a first job; A slicing unit, configured to slice the target job to obtain a set of job slices of the target job when it is detected that the execution of the first job is completed, wherein the set of job slices includes at least one job slice; A processing unit is used to process the at least one job slice through at least one computing node of the distributed system to complete the target job.

9. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 6 when running.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images