A method, apparatus, and device for batch scheduling jobs

Abstract

The present application discloses a method, apparatus, and device for batch scheduling jobs, which can be applied to the fields of artificial intelligence and finance, and can effectively improve the efficiency of bank batch scheduling jobs and solve the problem of insufficient resources of the current batch scheduling server. The method includes: First, import the jobs of file monitoring on the front-end page of the batch scheduling server so as to store the job information in the database, then receive the trigger operation on the front-end page of the scheduling server, execute the jobs of file monitoring, and form a request message with the job information of the jobs and send it to the scheduling client. Then, in response to the request message, the scheduling client polls and monitors whether the file exists in the corresponding path. If so, it will feedback the file arrival information to the server. Furthermore, according to the job name to be executed included in the request message, execute the corresponding job, and after the execution is completed, form a request message with the execution result and send it to the batch scheduling server to update the status of the job.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a method, device and equipment for batch scheduling jobs. Background Art

[0002] With the development of Internet finance, the data volume of traditional banking business systems implemented based on stored procedures is increasing, and the bank needs to handle more and more batch scheduling services.

[0003] However, in the existing bank scheduling system, the file-triggered batch jobs based on the server-side polling mode cause both thread overhead and CPU pressure on the scheduling server. Especially for the batch scheduling system facing all application products, if multiple products' file-triggered batch functions are handled simultaneously, the polling pressure is all on the batch scheduling server, which further increases the CPU overhead of the batch scheduling server and is prone to problems such as insufficient number of threads, thread blocking or long waiting. Therefore, how to improve the efficiency of bank batch scheduling jobs to solve the current problem of insufficient resources on the batch scheduling server is an urgent problem to be solved. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to provide a method, device and equipment for batch scheduling jobs, which can effectively improve the efficiency of bank batch scheduling jobs and solve the current problem of insufficient resources on the batch scheduling server.

[0005] In a first aspect, the embodiments of this application provide a method for batch scheduling jobs, and the method includes:

[0006] Import jobs for file monitoring on the front-end page of the batch scheduling server, so that the batch scheduling server stores the job information of the jobs in a database;

[0007] Receive a trigger operation on the front-end page of the scheduling server, execute the jobs for file monitoring, and form a request message with the job information of the jobs and send it to the scheduling client;

[0008] In response to the request message, the scheduling client polls and monitors whether a file exists in the corresponding path. If so, it will feedback the file arrival information to the server;

[0009] The scheduling client executes the corresponding jobs according to the job names to be executed included in the request message, and after the execution is completed, forms a request message with the execution result and sends it to the batch scheduling server, so that the server updates the status of the jobs according to the execution result and continues to execute the batch scheduling downward.

[0010] Optionally, in response to the request message, the scheduling client polls to monitor whether a file exists under the corresponding path. If so, it feeds back file arrival information to the server, including:

[0011] In response to the request message, the scheduling client passes the file path and file name included in the request message into the file polling module;

[0012] The file polling module starts a polling thread to monitor whether a file exists under the corresponding path. If not, it performs periodic polling detection until the file arrives at the path, then exits the polling thread and feeds back file arrival information to the server.

[0013] Optionally, the request message is an HTTP request message.

[0014] Optionally, the job information includes product name, process name, job name, file name, file path, and process structure information.

[0015] In a second aspect, an embodiment of the present application further provides an apparatus for batch scheduling jobs, the apparatus includes:

[0016] An import unit, configured to import jobs for file monitoring on the front-end page of the batch scheduling server, so that the batch scheduling server stores the job information of the jobs in a database;

[0017] A first sending unit, configured to receive a trigger operation on the front-end page of the scheduling server, execute the jobs for file monitoring, and form a request message with the job information of the jobs and send it to the scheduling client;

[0018] A feedback unit, configured that in response to the request message, the scheduling client polls to monitor whether a file exists under the corresponding path. If so, it feeds back file arrival information to the server;

[0019] A second sending unit, configured that the scheduling client executes corresponding jobs according to the job name to be executed included in the request message, and after the execution is completed, forms a request message with the execution result and sends it to the batch scheduling server, so that the server updates the status of the jobs according to the execution result and continues to execute the batch scheduling downward.

[0020] Optionally, the feedback unit includes:

[0021] An incoming sub-unit, configured that in response to the request message, the scheduling client passes the file path and file name included in the request message into the file polling module;

[0022] A feedback subunit, configured to start a polling thread through the file polling module to monitor whether a file exists in a corresponding path. If not, perform periodic polling detection until the file arrives at the path, then exit the polling thread and feedback the file arrival information to the server.

[0023] Optionally, the request message is an HTTP request message.

[0024] Optionally, the job information includes product name, process name, job name, file name, file path, and process structure information.

[0025] An embodiment of the present application further provides a device for batch scheduling jobs, including: a processor, a memory, and a system bus;

[0026] The processor and the memory are connected through the system bus;

[0027] The memory is used to store one or more programs, and the one or more programs include instructions that, when executed by the processor, cause the processor to execute any implementation manner of the above method for batch scheduling jobs.

[0028] An embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions run on a terminal device, the terminal device is caused to execute any implementation manner of the above method for batch scheduling jobs.

[0029] A method, device, and device for batch scheduling jobs provided by an embodiment of the present application. First, import a job for file monitoring on the front-end page of the batch scheduling server so that the batch scheduling server stores the job information of the job in a database. Then, receive a trigger operation on the front-end page of the scheduling server, execute the job for file monitoring, and form a request message with the job information of the job and send it to the scheduling client. Next, in response to the request message, the scheduling client polls and monitors whether a file exists in a corresponding path. If so, feedback the file arrival information to the server. Furthermore, the scheduling client executes a corresponding job according to the job name to be executed included in the request message, and after the execution is completed, forms a request message with the execution result and sends it to the batch scheduling server so that the server updates the status of the job according to the execution result and continues to perform batch scheduling downward. Thereby, the efficiency of bank batch scheduling jobs can be effectively improved, and the problem of insufficient resources of the current batch scheduling server can be solved. Description of the Drawings

[0030] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0031] Figure 1 It is a schematic flowchart of a method for batch scheduling jobs provided by an embodiment of the present application;

[0032] Figure 2 It is a schematic diagram of the composition of a device for batch scheduling jobs provided by an embodiment of the present application. Detailed implementation manners

[0033] In the current batch scheduling system of banks, there is often a batch scenario where a job in the batch scheduling process of a certain product needs to wait for the file of the upstream product to arrive before executing the batch job of this product. As for how to identify whether the file has reached the specified path, in the old scheduling system, the scheduling system server will start another asynchronous thread to poll and send HTTP requests to the target system (product application side). After the product application side receives the instruction, if the file has arrived, it will send the processing result to the batch scheduling server after processing the file. After the batch scheduling server updates the job processing result, it will complete the execution of the current job and continue to execute other jobs. If the file has not arrived, the product application side will return an instruction that the file has not arrived to the batch scheduling server, and the batch scheduling server will continue to execute the polling program and send requests to the application side periodically. Because in production, batch jobs often poll and wait before the upstream product file arrives, and the upstream product file generally arrives late, which makes the polling thread of the scheduling server execute for a long time. In addition, polling and sending requests to the application side to confirm messages, this polling mode of the scheduling server makes the thread overhead and CPU pressure on the scheduling server. Especially for a batch scheduling system that faces all application products, if it is dealing with the batch functions triggered by the files of multiple products at the same time, the polling pressure is on the scheduling server at the same time, which further increases the CPU overhead of the scheduling server and is prone to problems such as insufficient number of threads, thread blocking, or long waiting.

[0034] Therefore, how to improve the efficiency of bank batch scheduling jobs to solve the current problem of insufficient resources on the batch scheduling server is an urgent problem to be solved.

[0035] To address the above-mentioned deficiencies, an embodiment of the present application provides a method for batch scheduling jobs. First, jobs for file monitoring are imported on the front-end page of the batch scheduling server, so that the batch scheduling server stores the job information of the jobs in a database. Then, a trigger operation on the front-end page of the scheduling server is received, the jobs for file monitoring are executed, and the job information of the jobs is formed into a request message and sent to the scheduling client. Next, in response to the request message, the scheduling client polls to monitor whether the files in the corresponding path exist. If so, it will feedback the file arrival information to the server. Furthermore, the scheduling client executes the corresponding job according to the job name to be executed included in the request message. After the execution is completed, the execution result is formed into a request message and sent to the batch scheduling server, so that the server updates the status of the job according to the execution result and continues to execute the batch scheduling downward. This can effectively improve the efficiency of bank batch scheduling jobs and solve the problem of insufficient resources of the current batch scheduling server.

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0037] First Embodiment

[0038] See Figure 1 , which is a schematic flowchart of a method for batch scheduling jobs provided in this embodiment. The method includes the following steps:

[0039] S101: Import jobs for file monitoring on the front-end page of the batch scheduling server, so that the batch scheduling server stores the job information of the jobs in a database.

[0040] In this embodiment, in order to effectively improve the efficiency of bank batch scheduling jobs and solve the problem of insufficient resources of the current batch scheduling server, the present application proposes to optimize the file polling thread module and flexibly disassemble it into a batch scheduling client that can be relied on by each product application. When the batch scheduling system executes a process or job that requires file triggering, the file path and name are used as request parameters to send a request to the corresponding product application end. After the product application end receives the file polling request from the batch scheduling server, it starts a polling service in this application service to monitor whether the files in the corresponding path have been delivered. If they have been delivered, the polling service is ended and the file processing job is started. After the execution is completed, a completion request is sent to the batch scheduling server, and the batch scheduling server updates the status of the corresponding job and continues to execute downward.

[0041] In this application - side polling mode, the CPU overhead detected by the polling thread is dispersed across the application sides of each product, and the network communication between each product application side and the batch scheduling server is reduced to 2 times, greatly reducing the network overhead and solving the problem of insufficient resources on the batch scheduling server that may occur currently.

[0042] Specifically, first, the product user needs to import the file - monitoring job on the front - end page of the batch scheduling server so that the batch scheduling server stores the job information of the job in the database for subsequent execution of step S102.

[0043] Among them, the job information includes but is not limited to information such as product name, process name, job name, file name, file path, and process structure.

[0044] S102: Receive the trigger operation on the front - end page of the scheduling server, execute the file - monitoring job, and form a request message with the job information of the job and send it to the scheduling client.

[0045] In this embodiment, after importing the file - monitoring job on the front - end page of the batch scheduling server through step S101 so that the batch scheduling server stores the job information of the job in the database, further, it can receive the trigger operation on the front - end page of the scheduling server, execute the file - monitoring job, and form a request message with the job information of the job and send it to the scheduling client.

[0046] Among them, the request message can be an HTTP request message.

[0047] Specifically, the user can click the "Execute Process" operation on the front - end page of the scheduling server to start product batch scheduling. When it comes to the file - monitoring job, the scheduling server sets the current job status to "executing" and at the same time forms an HTTP request message with the job information data of the monitored file and sends it to the scheduling client, that is, the product application server.

[0048] S103: In response to the request message, the scheduling client polls to monitor whether the file exists in the corresponding path. If so, it will feedback the file arrival information to the server.

[0049] In this embodiment, after forming a request message with the job information of the job and sending it to the scheduling client through step S102, further, in response to the request message, the scheduling client polls to monitor whether the file exists in the corresponding path. If so, it will feedback the file arrival information to the server.

[0050] An optional implementation is as follows. First, in response to a request message, the scheduling client passes the file path and file name included in the request message to the file polling module. Then, the file polling module starts a polling thread to monitor whether the file exists in the corresponding path. If not, periodic polling detection is performed until the file arrives at the path, at which point the polling thread is exited and the file arrival information is fed back to the server.

[0051] Specifically, in this implementation, after the product application server receives a job request to monitor a file triggered by the scheduling server, it can pass the file path and file name to the file polling module. The file polling module then periodically starts a polling thread to monitor whether the file exists in the corresponding path. If the file does not exist, periodic polling detection is performed. Until the file arrives at the path, the polling thread is exited and the file arrival information is fed back to the application server.

[0052] S104: The scheduling client executes the corresponding job according to the job name to be executed included in the request message, and after the execution is completed, forms a request message with the execution result and sends it to the batch scheduling server, so that the server can update the status of the job according to the execution result and continue to execute the batch scheduling downward.

[0053] In this embodiment, after receiving the file arrival information, the scheduling client can further execute the corresponding job according to the job name to be executed included in the request message, start processing the file, and after the job execution is completed, send the execution result of the executed job (such as information on whether the job execution is successful or failed, execution duration, etc.) to the scheduling server through an HTTP request, so that the server can update the status of the job according to the execution result and continue to execute the batch scheduling downward.

[0054] In this way, by performing the above steps S101 - S104 for file-triggered batch jobs based on the application-side polling mode, the CPU overhead for the polling thread to detect files is dispersed among the application sides, and the network communication between each application side and the scheduling server is reduced to 2 times, greatly reducing the network overhead and solving the problem of insufficient resources of the batch scheduling server that may occur in the old mode.

[0055] For example: Suppose 50 application products simultaneously use the scheduling system to execute batch processes. Exactly when these 50 products simultaneously execute jobs that require file triggering, and the files required by these 50 products have not yet reached the specified path. At this time, the existing batch scheduling system will start 50 polling threads to send HTTP requests to each product to detect whether the files have been delivered. The pressure of 50 periodic polling threads and the sent HTTP requests is concentrated on the scheduling server. If each product polls 10 times, then 500 thread overheads and 1000 HTTP requests are required, which is extremely likely to cause insufficient idle threads and a large CPU overhead, resulting in the problem of insufficient resources on the scheduling server. However, the method for batch scheduling jobs provided in this application is based on the file-triggered batch job polling mode at the application end, so the polling overhead is dispersed among the 50 application products, only bearing the overhead of 100 HTTP requests, greatly reducing the CPU thread overhead and network overhead, and solving the problem of insufficient resources on the current batch scheduling server.

[0056] In summary, a method for batch scheduling jobs provided in this embodiment first imports file monitoring jobs on the front-end page of the batch scheduling server so that the batch scheduling server stores the job information of the jobs in the database. Then, it receives a trigger operation on the front-end page of the scheduling server, executes the file monitoring job, and forms a request message with the job information of the job and sends it to the scheduling client. Next, in response to the request message, the scheduling client polls to monitor whether the file exists in the corresponding path. If so, it will feedback the file arrival information to the server. Furthermore, the scheduling client executes the corresponding job according to the job name to be executed included in the request message, and after the execution is completed, forms a request message with the execution result and sends it to the batch scheduling server so that the server updates the status of the job according to the execution result and continues to execute the batch scheduling downward. Thus, it can effectively improve the efficiency of bank batch scheduling jobs and solve the problem of insufficient resources on the current batch scheduling server.

[0057] Second Embodiment

[0058] This embodiment will introduce a device for batch scheduling jobs. For related content, please refer to the above method embodiment.

[0059] See Figure 2 , which is a schematic diagram of the composition of a device for batch scheduling jobs provided in this embodiment, specifically including:

[0060] An import unit 201, configured to import file monitoring jobs on the front-end page of the batch scheduling server so that the batch scheduling server stores the job information of the jobs in the database;

[0061] The first sending unit 202 is configured to receive a trigger operation on the front-end page of the scheduling server, execute the job of the file monitoring, and form a request message with the job information of the job and send it to the scheduling client;

[0062] The feedback unit 203 is configured to, in response to the request message, the scheduling client polls and monitors whether a file exists in the corresponding path. If so, it feeds back the file arrival information to the server;

[0063] The second sending unit 204 is configured to, according to the job name to be executed included in the request message, the scheduling client executes the corresponding job, and after the execution is completed, form a request message with the execution result and send it to the batch scheduling server, so that the server updates the status of the job according to the execution result and continues to execute the scheduling batch downward.

[0064] In an implementation manner of this embodiment, the feedback unit 203 includes:

[0065] The incoming subunit is configured to, in response to the request message, the scheduling client passes the file path and file name included in the request message into the file polling module;

[0066] The feedback subunit is configured to start a polling thread through the file polling module to monitor whether a file exists in the corresponding path. If not, perform periodic polling detection until the file arrives at the path, then jump out of the polling thread, and feed back the file arrival information to the server.

[0067] In an implementation manner of this embodiment, the request message is an HTTP request message.

[0068] In an implementation manner of this embodiment, the job information includes product name, process name, job name, file name, file path, and process structure information.

[0069] In summary, a device for batch scheduling jobs provided in this embodiment first imports jobs for file monitoring on the front-end page of the batch scheduling server, so that the batch scheduling server stores the job information of the jobs in the database. Then, it receives a trigger operation on the front-end page of the scheduling server, executes the jobs for file monitoring, and forms a request message with the job information of the jobs and sends it to the scheduling client. Next, in response to the request message, the scheduling client polls and monitors whether the files exist in the corresponding path. If so, it will feedback the file arrival information to the server. Furthermore, the scheduling client executes the corresponding jobs according to the job names to be executed included in the request message, and after the execution is completed, it forms a request message with the execution result and sends it to the batch scheduling server, so that the server can update the status of the jobs according to the execution result and continue to execute the batch scheduling downward. Thereby, it can effectively improve the efficiency of bank batch scheduling jobs and solve the problem of insufficient resources of the current batch scheduling server.

[0070] Furthermore, an embodiment of the present application also provides a device for batch scheduling jobs, including: a processor, a memory, and a system bus;

[0071] The processor and the memory are connected through the system bus;

[0072] The memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes any implementation method of the above method for batch scheduling jobs.

[0073] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored, and when the instructions run on a terminal device, the terminal device executes any implementation method of the above method for batch scheduling jobs.

[0074] From the description of the above embodiments, those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment or some parts of the embodiments of the present application.

[0075] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0076] It should also be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0077] In addition, it should be noted that the method, device and equipment for batch scheduling jobs provided by the present invention can be used in the field of artificial intelligence and the financial field. The above are only examples and do not limit the application fields of the method, device and equipment for batch scheduling jobs provided by the present invention.

[0078] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for batch scheduling jobs, characterized in that, the method includes: Importing jobs for file monitoring on the front-end page of the batch scheduling server, so that the batch scheduling server stores the job information of the jobs in a database; Receiving a trigger operation on the front-end page of the scheduling server, executing the jobs for file monitoring, and forming a request message with the job information of the jobs and sending it to the scheduling client; In response to the request message, the scheduling client polls and monitors whether the file exists in the corresponding path. If so, it will feedback the file arrival information to the server; The scheduling client executes the corresponding job according to the job name to be executed included in the request message, and after the execution is completed, forms a request message with the execution result and sends it to the batch scheduling server, so that the server updates the status of the job according to the execution result and continues to execute the batch scheduling downward.

2. The method according to claim 1, characterized in that, In response to the request message, the scheduling client polls and monitors whether the file exists in the corresponding path. If so, it will feedback the file arrival information to the server, including: In response to the request message, the scheduling client passes the file path and file name included in the request message into the file polling module; The file polling module starts a polling thread to monitor whether the file exists in the corresponding path. If not, it performs periodic polling detection until the file reaches the path, then exits the polling thread and feedbacks the file arrival information to the server.

3. The method according to claim 1, characterized in that, The request message is an HTTP request message.

4. The method according to any one of claims 1-3, characterized in that, The job information includes product name, process name, job name, file name, file path, and process structure information.

5. An apparatus for batch scheduling jobs, characterized in that, the apparatus includes: An import unit for importing jobs for file monitoring on the front-end page of the batch scheduling server, so that the batch scheduling server stores the job information of the jobs in a database; A first sending unit for receiving a trigger operation on the front-end page of the scheduling server, executing the jobs for file monitoring, and forming a request message with the job information of the jobs and sending it to the scheduling client; A feedback unit for, in response to the request message, the scheduling client polling and monitoring whether the file exists in the corresponding path. If so, it will feedback the file arrival information to the server; A second sending unit for the scheduling client to execute the corresponding job according to the job name to be executed included in the request message, and after the execution is completed, form a request message with the execution result and send it to the batch scheduling server, so that the server updates the status of the job according to the execution result and continues to execute the batch scheduling downward.

6. The apparatus according to claim 5, characterized in that, The feedback unit includes: An incoming subunit, configured to, in response to the request message, the scheduling client transmits the file path and file name included in the request message to the file polling module; A feedback subunit, configured to start a polling thread through the file polling module to monitor whether a file exists in the corresponding path. If not, perform periodic polling detection until the file arrives at the path, then exit the polling thread, and feedback the file arrival information to the server.

7. The apparatus according to claim 5, wherein, the request message is an HTTP request message.

8. The apparatus according to any one of claims 5-7, wherein, the job information includes product name, process name, job name, file name, file path, and process structure information.

9. An apparatus for batch scheduling jobs, wherein, comprises: a processor, a memory, and a system bus; the processor and the memory are connected through the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1-4.

10. A computer-readable storage medium, wherein, the computer-readable storage medium stores instructions, and when the instructions run on a terminal device, the terminal device executes the method according to any one of claims 1-4.

Images