Distributed data computing framework job submission method, device and equipment

Abstract

The application provides a distributed data computing framework job submission method, device and equipment, and is used for solving the technical problem of how to efficiently submit a distributed data computing framework job in a distributed system. The application records job information in a job database, and a job submission client on one or more nodes reads a job meeting state and queue conditions from the job database and submits the job to a distributed resource manager for scheduling and running. In the case that a node or network failure causes the job submission to fail, the application can automatically recover the job through a fault recovery process. The application can dynamically adjust the job submission capability according to the resources and load of the node where the job submission client is located, improve the adaptability and load balancing of the job submission, and implement parallel job submission through multiple job submission clients on multiple nodes to improve the job submission efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication and big data technology, and particularly relates to a distributed data computing framework job submission method, device and equipment. BACKGROUND

[0002] Apache Flink is a distributed data computing framework, which is used for stateful computation on unbounded and bounded data streams. Flink can run in all common cluster environments.

[0003] Apache Hadoop YARN is a resource manager in Hadoop distributed system infrastructure ecology, which is a general resource management system and can provide unified resource management and scheduling for upper-layer applications. The basic idea of YARN is to separate the functions of resource management and job scheduling / monitoring into separate daemons. The main method is to create a global resource manager (Resource Manager, RM) and a plurality of application masters (ApplicationMaster, AM) for applications. The application can be a single job or a DAG (Directed Acyclic Graph) of jobs.

[0004] With the wide popularity of big data technology and the increasing requirements of real-time performance in more and more application scenarios, Flink, as a distributed stream batch data processing framework software, is widely used, and many businesses begin to use Flink as a data processing engine. However, with the increase of Flink application programs (also referred to as Flink jobs), challenges are also brought to the submission of Flink jobs.

[0005] In a large data processing process, a plurality of Flink application programs can be included. After the user develops the Flink application programs, a batch of Flink application programs is submitted to Yarn for scheduling and running. The submission of a plurality of Flink application programs brings great pressure to the job submission end, and the speed of single node submission cannot meet the requirements.

[0006] Flink as a stream processing framework software, it provides a set of mechanism that can submit Flink job through Shell script (located in the Flink installation directory bin directory) itself. Users can call the script to submit the job. For the shell script submission mode, the same node needs to open multiple terminals at the same time, switch execution, to achieve parallel submission. If the submission fails, or the submission node abnormally, only manual comparison can determine which job is successfully submitted. At the same time, there is no job content verification mechanism, and the job runs failure after submission, and if all jobs are submitted on one node, it will inevitably increase the resource consumption of the node and affect the running of other programs on the node. SUMMARY

[0007] Therefore, the present application provides a distributed data computing framework job submission method, device and equipment, which solves the technical problem of how to efficiently submit a distributed data computing framework job in a distributed system.

[0008] Based on one aspect of the embodiments of the present application, the present application provides a distributed data computing framework job submission method, which comprises:

[0009] When receiving a job submission instruction of a specified job, marking the job record of the specified job in the job database as a submitting state;

[0010] Applying for resources for running the specified job to a resource manager in a distributed system infrastructure;

[0011] When the resource application is successful, obtaining a job logic diagram of the specified job, and submitting the specified job to the resource manager for scheduling and running;

[0012] When the resource application fails, marking the job record of the specified job in the job database as a waiting state to wait for re-submission.

[0013] Further, the method for re-submitting the specified job in the waiting state is:

[0014] When the scanning period arrives, obtaining a mutual exclusion lock for controlling reading the job in the waiting state in the job database;

[0015] Judging whether the mutual exclusion lock acquisition is successful;

[0016] When the mutual exclusion lock acquisition is successful, obtaining the job in the waiting state in the job database and trying to re-submit.

[0017] Further, the method further comprises the step of controlling the order and quantity of submitting jobs by each node through a job queue, and the specific method is:

[0018] When receiving a job submission instruction of a specified job, first, the number of idle queue items in the local job queue is judged, if the queue is full, the specified job in the job database is marked as a waiting state, and the current job submission operation is abandoned; if the queue is not full, the job record of the specified job in the job database is marked as a submission state, and then the specified job is entered into the job queue, and the jobs in the job queue are submitted in a first-in first-out manner.

[0019] Further, when the scanning period is reached, first, the number of idle queue items in the local job queue is judged, if the queue is full, the operation of obtaining the job in the waiting state is abandoned; if the queue is not full, the number of idle queue items is taken as a limit condition to obtain the job in the waiting state from the job database, and the number of idle queue items is not greater than the number of idle queue items, after the job is obtained, the job is added to the job queue, and the jobs in the job queue are submitted in a first-in first-out manner.

[0020] Further, the invalidation time parameter is set for the mutual exclusion lock, if the node does not release the mutual exclusion lock after the invalidation time of the mutual exclusion lock, the node is forced to end the occupation of the mutual exclusion lock, and the job submission process of the node is terminated.

[0021] Further, the method further includes an abnormal recovery processing process, the abnormal recovery processing process is executed when the node is restarted, and the process steps include:

[0022] obtaining the job in the submission state from the job database;

[0023] obtaining the running record of the obtained job in the submission state from the resource manager;

[0024] if the running record is obtained, the job state of the corresponding job in the job database is updated according to the running record;

[0025] if the running record is not obtained, the submission processing flow of the corresponding job is executed again.

[0026] Further, the normal submission switch parameter is set to the closed state in the execution of the abnormal recovery processing process; in the closed state, the normal job submission processing process of the current node is limited.

[0027] Based on another aspect of the embodiment of the application, the application further provides a distributed data computing framework job submission device, the device includes:

[0028] an information maintenance module, configured to mark the job record of the specified job in the job database as a submission state when receiving a job submission instruction of a specified job;

[0029] a resource application module, configured to apply for resources for running the specified job from a resource manager in a distributed system infrastructure;

[0030] a job submission module, configured to acquire a job logic diagram of the specified job and submit the specified job to the resource manager for scheduling and running when the resource application is successful;

[0031] The information maintenance module is further configured to mark a job record of the specified job in the job database as a waiting state to wait for re-submission when the resource application fails.

[0032] Further, the job submission module is further configured to periodically scan jobs in the waiting state in the job database, acquire a mutual exclusion lock for controlling reading of the jobs in the waiting state in the job database when a scanning period arrives, judge whether the mutual exclusion lock acquisition is successful, acquire the jobs in the waiting state in the job database and attempt re-submission when the mutual exclusion lock acquisition is successful.

[0033] The mutual exclusion lock has a time limit, and if the node has not released the mutual exclusion lock when the mutual exclusion lock is invalid, the node is forced to end the occupation of the mutual exclusion lock and terminate the job submission process of the node.

[0034] Further, the device further comprises:

[0035] a job queue module, configured to store jobs waiting for submission; the job queue is used to control the order and quantity of submission of jobs by each node;

[0036] The job submission module is configured to, when receiving a job submission instruction of a specified job, first judge the number of idle queue items in the local job queue, mark the specified job in the job database as a waiting state and give up the job submission operation if the queue is full, mark a job record of the specified job in the job database as a submission state and then enter the specified job into the job queue if the queue is not full, and submit jobs in the job queue in a first-in first-out manner.

[0037] The job submission module is configured to, when the scanning period arrives, first judge the number of idle queue items in the local job queue, give up the operation of acquiring jobs in the waiting state if the queue is full, and acquire jobs in the waiting state from the job database with the number of idle queue items as a limit if the queue is not full, and then add the jobs to the job queue and submit jobs in the job queue in a first-in first-out manner.

[0038] Further, the device further comprises:

[0039] an abnormality recovery module, configured to perform an abnormality recovery process when a node is restarted, and the module comprises:

[0040] a job obtaining unit, configured to obtain a job in a submitting state belonging to the current node from a job database;

[0041] a running record obtaining unit, configured to obtain a running record of the obtained job in the submitting state from a resource manager to determine whether the running record of the job in the submitting state exists on the resource manager;

[0042] an exception recovery processing unit, configured to update a job state of the corresponding job in the job database according to the running record when the running record is obtained, and execute a submitting processing procedure of the corresponding job again when the running record is not obtained.

[0043] The apparatus described in the present application can be realized in the form of software, hardware or a combination of software and hardware. When realized in the form of software modules, the program codes of the software modules are loaded into a storage medium of a device, the program codes in the storage medium are read by a processor and executed, so as to realize the functions of the constituent modules in the apparatus. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0045] Figure 1 a flowchart of a job submitting method of a distributed data computing framework provided by an embodiment of the present application;

[0046] Figure 2 a flowchart of a step of periodically scanning a job database to submit a job in a waiting state in an embodiment of the present application;

[0047] Figure 3 a flowchart of a step of an exception recovery method provided by an embodiment of the present application;

[0048] Figure 4 a structural schematic diagram of an electronic device for realizing the job submitting method of the distributed data computing framework provided by the present application. DETAILED DESCRIPTION

[0049] The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the present application. As used in the description of the embodiments of the present application and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, objects, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, and / or components thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the term "if' can be construed to mean "when" or "upon" or "in response to determining" or some similar temporal term, and not a conclusive condition. The term "and / or" used in the present application refers to the association relationship of the associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, in the description of the present application, "a plurality of" means two or more than two, unless otherwise specified. "At least one of" or the like refers to any combination of the items, including a single item or a combination of multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be singular or plural.

[0050] The purpose of the present application is to provide a technical solution for submitting a job of a distributed data computing framework (such as Flink) to a distributed resource manager (such as Yarn) for scheduling and running the job by the distributed resource manager. The basic idea of the technical solution is to record job information including job status in a job database, and to submit a job that meets the state and queue conditions from the job database to the resource manager for scheduling and running by a job submission client on one or more nodes. The technical solution records job status information in a database, and in the case of job submission failure due to node failure or network failure, the job can be automatically recovered through a fault recovery process. The job submission capability can be dynamically adjusted according to the resources and load of the node where the job submission client is located, improving the adaptability and load balancing of job submission. Multiple job submission clients on multiple nodes can be used to submit jobs in parallel, improving the efficiency of job submission.

[0051] Based on the basic idea of the present application, the specific implementation process of the present application will be described below in combination with the drawings and specific embodiments.

[0052] Figure 1A flowchart of a job submission method of a distributed data computing framework provided by an embodiment of the present application is shown. The method is applied to one or more devices that submit an application for completing a job task to a distributed data computing framework (such as Flink). The method comprises the following steps:

[0053] Step 101: When a job submission instruction of a specified job is received, mark the job record of the specified job in the job database as a submitting state;

[0054] The Flink job can be generated by a big data management platform. The job information can be recorded in the job database when the job is generated. The job database can be connected and shared with multiple job submission clients located on different nodes.

[0055] When a job submission instruction of a specified job is received by the job submission client / job submission device located on a certain node, the job state of the specified job in the job database can be modified to a submitting state first, so as to identify that the job is being submitted to the resource manager.

[0056] In the present application, the job database is used to record the job information of all jobs. Each record in the job database corresponds to a job task. Taking Flink as an example, each Flink job is completed by submitting a Flink application to Yarn. Each Flink job corresponds to a job record in the job database. The job information included in each job record at least includes: job name, job state.

[0057] The persistent data structure used to record the job information in the present application is not limited to the form of a database. It can also be a configuration file located in a shared location, a persistent memory data structure object, etc.

[0058] Step 102: Apply for resources for running the specified job to a resource manager (such as Yarn) in a distributed system infrastructure (such as Hadoop);

[0059] The resource manager Yarn in the distributed system infrastructure Hadoop is used to schedule and run Flink jobs. Before submitting a Flink job to Yarn, resources for running the Flink application for completing the job need to be applied to Yarn first. The job can be submitted to Yarn only when the application is successful. When the resource application fails, the job needs to be placed in a waiting state. Subsequently, the job in the waiting state is executed again by a timing task for scanning and submitting.

[0060] Step 103: When the resource application is successful, obtain the job logic graph (such as JobGraph) of the specified job, and submit the specified job to the resource manager for scheduling and running;

[0061] JobGraph is a job logic graph of a Flink application, which is a Flink internal data structure. When a Flink job is submitted to Yarn, the Flink job is submitted to Yarn in the form of JobGraph, and the Flink job is scheduled and run on Yarn.

[0062] In this step, the way of obtaining the job logic graph of the specified job can be to generate the job logic graph in real time in this step, or to obtain the job logic graph generated and cached in the job development stage. Flink provides a Yarn Application mode, which can first apply for running resources to Yarn, and then build a JobGraph object and run after the resources are ready.

[0063] After the job logic graph is generated in the job development stage, the generated job logic graph can be verified. For example, according to the API specification requirements of Flink, it is verified whether the Flink JobGraph data structure can be generated and whether it meets the specification requirements. In this way, it can be avoided that the job logic graph has content or logical errors when the job is submitted and run, thereby improving the efficiency and correctness of job submission.

[0064] After the specified job is successfully submitted to Yarn, the state of the specified job in the job database is modified to the running state RUNNING, to indicate that the job is successfully submitted and is in the running state.

[0065] If an exception occurs in the job submission process and does not cause the job submission client program to crash, the state of the specified job in the job database is modified to the submission failed state SUBMIT_FAILED, to indicate that the job submission fails. The job that fails to submit can be recovered in the subsequent exception recovery processing step and tried to be submitted again.

[0066] Step 104: When the resource application fails, the job record of the specified job in the job database is marked as a waiting state to wait for re-submission.

[0067] When the resources of Yarn are insufficient to schedule or run the job, the job submission client may fail to apply for resources from Yarn. In order to improve the intelligence and reliability of job submission, the present application triggers the re-submission of the job that fails to be submitted due to insufficient resources through a periodic scanning process.

[0068] Figure 2The step flow diagram for submitting the job in the waiting state in the job database in the periodic scanning job in an embodiment of the present application. In this embodiment, a mutual exclusion lock is set for the step of obtaining the job in the waiting state in the job database. The purpose of setting the mutual exclusion lock is to prevent multiple nodes from repeatedly obtaining the job in the waiting state, resulting in repeated submission of the job. Each job submission node automatically scans the job database through a timing task, and reads the job in the waiting state from the database. The specific steps are as follows:

[0069] Step 201: When the scanning period arrives, obtain the mutual exclusion lock for controlling reading of the job in the waiting state in the job database.

[0070] In a distributed scenario, the mutual exclusion lock can be implemented using a remote key-value memory database such as Redis.

[0071] Step 202: Determine whether the mutual exclusion lock is successfully obtained.

[0072] Step 203: When the mutual exclusion lock is successfully obtained, obtain the job in the waiting state in the job database and attempt to submit it again.

[0073] The node that obtains the mutual exclusion lock reads the job in the waiting state from the job database. If one or more jobs in the waiting state are obtained, the state of the obtained job in the waiting state is set to SUBMITTING, and then the processing step of step 102 is executed again.

[0074] In another embodiment of the present application, in order to avoid a node obtaining too many jobs in the waiting state, which occupies too many node resources or causes a backlog of jobs to be submitted, a job submission queue is set for the job submission client to control the order and quantity of job submission. The queue depth of the job queue can be configured, and the queue depth of the job queue limits the maximum number of jobs in the waiting state obtained by the node from the job database each time. Limiting the number of jobs in the waiting state obtained by a single node can also improve the efficiency of parallel job submission by multiple nodes.

[0075] When the node obtains the job in the waiting state from the job database, it first determines the number of idle queue items in the local job queue. If the queue is full, the node will not obtain the job in the waiting state from the job database. If the queue is not full, the number of idle queue items is used as a limit to obtain jobs in the waiting state from the job database, and the jobs are added to the job queue after being obtained. The node can submit the jobs in the queue in the order in which they were entered.

[0076] Further, in order to avoid the situation that a node cannot obtain a job in a waiting state from the job database due to the fact that the node cannot obtain a mutual exclusion lock because the node has not released the mutual exclusion lock for a long time after obtaining the mutual exclusion lock, an embodiment of the present application sets a time-to-live parameter for the mutual exclusion lock. If a node scans a job in a waiting (WAIT) state and has added the job to a job queue, the node will normally release the mutual exclusion lock within the time-to-live of the mutual exclusion lock. However, if the node has not released the mutual exclusion lock after the time-to-live of the mutual exclusion lock, the node is forced to end the occupation of the mutual exclusion lock and terminate the job submission process of the node, and the job state is set to a submission failure state. Adding the time-to-live to the mutual exclusion lock can effectively avoid the situation that a job submission client that has occupied the mutual exclusion lock cannot release the mutual exclusion lock due to an abnormality (a fatal error of a program or a power failure of a machine, etc.) of the job submission client, thereby blocking other normally working submission program nodes from scanning a job in a WAIT state in the database.

[0077] In another embodiment of the present application, in order to avoid an abnormality (for example, a program crash, a power failure, etc.) of a job submission client during a job submission process, the job submission method provided by the present application further includes a processing step of abnormality recovery. The abnormality recovery process of a job is only executed when a program is started. In the case of a power failure, the state of the job can be in a submitting (SUBMITTING) state. In the case of a program crash or some abnormality of a Yarn end, the state of the job can be in a submission failure (SUBMIT_FAILED) state. In order to handle the abnormality, when a job submission client is restarted, an abnormality recovery process can be executed to repair the inconsistency between a submission end and a resource Yarn end due to the abnormality.

[0078] Figure 3 For the step flowchart of the abnormality recovery method provided by an embodiment of the present application, the abnormality recovery flowchart is provided with a normal submission switch parameter. When the abnormality recovery flowchart is entered, the normal submission switch is initially in a closed state, so as to avoid a conflict between a normal submission process and an abnormality recovery process. The abnormality recovery process specifically includes:

[0079] Step 301: obtaining a job in a submitting state from a job database and belonging to a current node;

[0080] After a job submission client is restarted and enters an abnormality recovery process, the normal submission switch is set to a closed state to limit the current node from processing a normal job submission process until the abnormality recovery process is completed, so as to avoid a conflict between a normal job submission process and an abnormality recovery process.

[0081] If the current node receives a normal job submission instruction during the abnormal recovery process, the job state can be directly set to the waiting state, and the job is put into the job database for submission by other normal nodes, so as to avoid the normal job submission process from being stuck in the current node.

[0082] Table 1 is an example of job information in the job database. The job submission client generates a unique submission tag (Tag) every time it is restarted, which is written into the database together with the job state changed to SUBMITTING. The Tag prevents repeated submission when the submission program is restarted.

[0083] Table 1

[0084] Job number Job name Job status Submit node identification Submit tag Application identification 1 JOB1 WAIT A 2 JOB2 SUBMITTING B 2A1233334 3 JOB3 SUBMIT_FAILED B 2A1233334

[0085] During the abnormal recovery process, the job database is searched for jobs that may have submission abnormalities by using the submission node identifier and the job state. For example, if node B has an abnormality, after node B is restarted, the job states SUBMITTING and SUBMIT_FAILED can be used to search for JOB2 and JOB3, which may need to be recovered.

[0086] For jobs in the SUBMITTING state, the job states of the submission end and the resource manager end need to be compared, and the states of the two ends are synchronized according to the actual submission result of the job.

[0087] For jobs that have failed to submit, the node can attempt to submit again a limited number of times and record the log records of the submission process.

[0088] Step 302: Obtain the running record of the job in the submitting state from the resource manager to determine whether the running record of the job exists on the resource manager.

[0089] The job submission client connects to Yarn to obtain the running record information of the Flink job on Yarn. According to the job name and the submission tag Tag, it is determined whether the job has been submitted to Yarn before the program abnormality. If it has been submitted, the submission result is determined according to the job state on Yarn. If the submission is not successful, a new Tag is set for the job in the job database (in order to ensure that each execution of the submission has a new Tag, the same job may be repeatedly online and offline, and therefore the Tag is used as the identifier of each submission execution) for re-execution of the submission.

[0090] Step 303: If the running record is obtained, the job state of the corresponding job in the job database is updated according to the running record.

[0091] For jobs submitted to Yarn and already running on Yarn, a running record is kept in the resource manager. This record includes information such as the application identifier (Yarn Application ID) generated for the job, the job's running status, and the results. After obtaining the job's running record, the node that submitted the job can update the job status in the job database based on the running record. For example, if the job was successfully submitted and is running normally, the job status will be updated to RUNNING.

[0092] The application identifier refers to the unique identifier (Yarn Application ID) assigned to each Flink job after it is submitted to Yarn and runs as an application within Yarn. This value is stored in the job information database and can provide a basis for obtaining monitoring information of jobs running on Yarn for extended periods (the job's running information can be obtained through the Yarn Application ID, such as whether the job is still running, whether it has terminated abnormally, and the Yarn running logs, etc.).

[0093] Step 304: If the running record is not obtained, the corresponding job submission process is executed again.

[0094] The submission processing flow refers to Figure 1 The example shows the processing steps 102 to 104.

[0095] After completing all the abnormal recovery processing steps for the acquired jobs in the abnormal recovery process, the normal submission switch is turned on so that the job submission client can enter the normal job submission process.

[0096] The technical solution provided in this application records job information through a job database. If a submitted job cannot be executed immediately, its information is stored in the job database and submitted in a first-in, first-out (FIFO) queue, thus achieving load balancing among the submission program nodes. If a node malfunctions and the job submission client program crashes, upon recovery, the program verifies the previously malfunctioning jobs. Successfully submitted jobs are assigned their running information on Yarn, while unsuccessfully submitted jobs are resubmitted, thereby achieving submission recovery and disaster recovery capabilities. The technical solution provided in this application allows for parallel submission across multiple nodes. If the number of jobs is large, more nodes can be added to enhance the job submission capacity.

[0097] Figure 4This is a schematic diagram of an electronic device structure for implementing the job submission method of the distributed data computing framework provided in this application, according to an embodiment of the present application. The device 400 includes: a processor 410 such as a central processing unit (CPU), a communication bus 420, a communication interface 440, and a memory 430. The processor 410 and the memory 430 can communicate with each other via the communication bus 420. The memory 430 stores a computer program, which, when executed by the processor 410, can implement the functions of one or more steps in the job submission method of the distributed data computing framework provided in this application.

[0098] Memory refers to a device based on a storage medium used to store computer programs and / or data. It can be volatile memory (VM, often called main memory) or non-volatile memory (NVM). Main memory refers to internal storage that directly exchanges data with the processor. It can read and write data at any time and at high speed, serving as a storage medium for temporary data of the operating system and other running programs. Main memory can be Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), etc. Non-volatile memory refers to memory that uses persistent storage media, characterized by large capacity and persistent data retention. It can be Storage Class Memory (SCM), Solid State Drive (SSD), NAND flash memory, hard disks, etc. SCM is an industry term for a new type of storage medium that falls between main memory and flash memory. It is a composite storage technology that combines the characteristics of persistent storage and main memory, with access speeds slower than DRAM but faster than SSDs.

[0099] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0100] It should be understood that embodiments of this application can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory (or non-persistent) memory. The methods can be implemented using standard programming techniques, including a non-transitory storage medium configured with a computer program within the computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit. Furthermore, the operations of the processes described in this application can be performed in any suitable order unless otherwise indicated by this application or otherwise clearly contradicted by the context. The processes described in this application (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0101] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices. Aspects of this application can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, which, when read by the computer, can be used to configure and operate the computer to perform the processes described herein. Additionally, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described in this application includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. This application also includes the computer itself when programmed according to the methods and techniques described herein.

[0102] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A distributed data computing framework job submission method, characterized in that, The method comprises: When a job submission instruction of a specified job is received, marking a job record of the specified job in a job database as a submitting state; Applying for a resource for running the specified job to a resource manager in a distributed system infrastructure; When the resource application is successful, obtaining a job logic diagram of the specified job, and submitting the specified job to the resource manager for scheduling and running; When the resource application fails, marking the job record of the specified job in the job database as a waiting state to wait for re-submission; The method further comprises an exception recovery process which is executed when a node is restarted, and the process steps comprise: Obtaining a job in a submitting state from the job database which belongs to the current node; Obtaining a running record of the job in the submitting state from the resource manager to determine whether the running record exists on the resource manager; If the running record is obtained, updating a job state of the corresponding job in the job database according to the running record; If the running record is not obtained, executing a submission processing flow of the corresponding job again.

2. The method of claim 1, wherein, The method for re-submitting the specified job in the waiting state comprises: When a scanning period arrives, obtaining a mutual exclusion lock for controlling reading of the job in the waiting state in the job database; Determining whether the mutual exclusion lock is obtained successfully; When the mutual exclusion lock is obtained successfully, obtaining the job in the waiting state in the job database and attempting to re-submit.

3. The method of claim 2, wherein, The method further comprises a step of controlling a sequence and a quantity of jobs submitted by each node through a job queue, and the specific method comprises: When a job submission instruction of a specified job is received, first determining a number of idle queue items in a local job queue, if the queue is full, marking the specified job in the job database as a waiting state, and abandoning the job submission operation this time; If the queue is not full, marking a job record of the specified job in the job database as a submitting state, and then entering the specified job into the job queue to submit the jobs in the job queue in a first-in first-out manner.

4. The method according to claim 3, wherein, When a scanning period arrives, first determining a number of idle queue items in a local job queue, if the queue is full, abandoning the operation of obtaining the job in the waiting state this time; if the queue is not full, obtaining the job in the waiting state from the job database as a limit condition of the number of idle queue items, and then adding the job into the job queue to submit the jobs in the job queue in a first-in first-out manner.

5. The method of claim 4, wherein, A time-out parameter is set for the mutual exclusion lock, if the node does not release the mutual exclusion lock after the time-out of the mutual exclusion lock, the node is forced to end the occupation of the mutual exclusion lock, and the job submission process of the node is terminated.

6. The method according to claim 1, wherein, During the execution of the exception recovery process, a normal submission switch parameter is set as a closed state; The normal submission switch limits the submission processing flow of the normal job handled by the current node in the closed state.

7. A distributed data computing framework job submission apparatus, characterized by comprising: The device comprises: The information maintenance module is configured to mark a job record of a specified job in the job database as a submitting state when a job submission instruction of the specified job is received; The resource application module is configured to apply for resources for running the specified job from a resource manager in a distributed system infrastructure; The job submission module is configured to obtain a job logic diagram of the specified job and submit the specified job to the resource manager for scheduling and running when the resource application is successful; The information maintenance module is further configured to mark the job record of the specified job in the job database as a waiting state to wait for re-submission when the resource application fails; The abnormal recovery module is configured to perform an abnormal recovery process when a node is restarted, and the module comprises: A job obtaining unit is configured to obtain a job in a submitting state belonging to a current node from the job database; A running record obtaining unit is configured to obtain a running record of the job in the submitting state from the resource manager to determine whether the running record exists on the resource manager; An abnormal recovery processing unit is configured to update a job state of a corresponding job in the job database according to the running record when the running record is obtained, and perform a submission processing procedure of the corresponding job again when the running record is not obtained.

8. The apparatus of claim 7, wherein The job submission module is further configured to periodically scan jobs in a waiting state in the job database, obtain a mutual exclusion lock for controlling reading of the jobs in the waiting state in the job database when a scanning period arrives, determine whether the mutual exclusion lock is successfully obtained, and obtain the jobs in the waiting state in the job database and attempt to re-submit when the mutual exclusion lock is successfully obtained; The mutual exclusion lock has a time limit, and if the node has not released the mutual exclusion lock when the mutual exclusion lock expires, the node is forced to end occupation of the mutual exclusion lock and terminate the job submission process of the node.

9. The apparatus of claim 8, wherein, The apparatus further comprises: A job queue module is configured to store jobs waiting for submission, and the job queue is used to control a sequence and a quantity of jobs submitted by each node; The job submission module is configured to determine a number of idle queue items in a local job queue when a job submission instruction of a specified job is received, mark the specified job in the job database as a waiting state and give up the job submission operation if the queue is full, mark a job record of the specified job in the job database as a submitting state and enter the specified job into the job queue if the queue is not full, and submit jobs in the job queue in a first-in-first-out manner; The job submission module is configured to determine a number of idle queue items in a local job queue when a scanning period arrives, give up the operation of obtaining jobs in a waiting state if the queue is full, obtain jobs in the waiting state from the job database as a limit condition of the number of idle queue items if the queue is not full, and add the jobs into the job queue and submit jobs in the job queue in a first-in-first-out manner.

10. An electronic device, comprising: The apparatus comprises a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium complete communication with each other through the communication bus. - a storage medium for depositing a computer program, - a processor for implementing the method steps of any one of claims 1-6 when executing the computer program deposited on the storage medium.

11. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the method of any one of claims 1 to 6.

Images