A method and system for proxy client parallel scheduling

Abstract

The present application relates to the technical field of communication, and especially relates to a method and system for parallel scheduling based on proxy client. The method comprises the following steps: a scheduling system server calls an official client server for batch jobs through a client; a mapping relationship between the official client server and a proxy client server is constructed; the official client server calls the proxy client server for batch jobs through a proxy client sending end; the proxy client server synchronously executes results to the official client server through a proxy client receiving end; and the official client server sends the local execution results and the proxy client server execution results to the scheduling system server. The method is suitable for the case that the target host cannot install the client of the scheduling management system, and simultaneously schedules different controlled hosts through the proxy client, so as to realize the purpose of synchronously scheduling different host jobs in parallel, and the development cost is small and the cycle is short compared with developing a new client.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method and system for parallel scheduling based on proxy clients. Background Technology

[0002] In the National High-Tech R&D Program (863 Program), the host, database, storage, and scheduling management systems that completely replace batch module execution must be replaced using domestically developed and controllable hardware and software. During the production and development / testing phases, due to objective conditions, there is a need to selectively compare or eliminate differences in certain aspects, sometimes requiring the replacement of only the host, database, storage, or management system. For example, to shield against the influence of other factors, it may be necessary to replace only the storage and test its performance; or to mask differences in storage, maintain identical storage, and replace other hosts, databases, and scheduling management systems.

[0003] The batch processing module uses a large number of dependent shell scripts for system processing. Its core is a job scheduling and management system. Different scheduling and management systems vary greatly in their implementation, data structures, and scheduling logic. Replacing one can significantly impact the details of the batch module's execution, such as the time it takes for the controlled host to return the result to the server management end after a job succeeds, whether logical job associations are needed between jobs, the efficiency of the management end in checking jobs that meet the conditions and sending them to the controlled hosts for execution, and the load on each controlled host during the scheduling process.

[0004] To eliminate the differences between scheduling management systems, these differences need to be masked during the adaptation process of the National High-Tech R&D Program (NTDP). The two environments (controlled hosts, databases, storage, etc.) are compared and tested using the scheduling management systems before and after the replacement to ensure compatibility. However, clients are generally developed and installed based on the controlled hosts. For example, scheduling management systems A and B each support different controlled hosts a and b. Often, a and b are not the same. If the scheduling system can only control its associated controlled host system, the database and storage may also depend on the host system, further widening the differences to ensure normal operation. This fails to achieve the goal of eliminating only the differences between the scheduling management systems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method and system for parallel scheduling based on agent clients.

[0006] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0007] A method for parallel scheduling based on a proxy client, characterized by the following steps:

[0008] Step S1: The scheduling system server calls the official client server through the client to perform batch jobs;

[0009] Step S2: Construct a mapping relationship between the official client server and the proxy client server, wherein the mapping relationship includes one-to-one mapping, many-to-one mapping, or one-to-many mapping;

[0010] Step S3: The official client server calls the proxy client server to perform batch jobs through the proxy client sending end using parallel job instructions;

[0011] Step S4: The proxy client server synchronizes the execution results to the official client server through the proxy client receiving end;

[0012] Step S5: The official client server combines the execution results from the local machine and the execution results from the proxy client server and sends them to the scheduling system server.

[0013] Furthermore, the proxy client server includes N levels of proxy client servers, where N is greater than or equal to 1.

[0014] Furthermore, the proxy client server includes a receiving end and a sending end, and the upper-level proxy client server calls the lower-level proxy client server to perform batch operations through the sending end.

[0015] Furthermore, step S3 also includes the following sub-steps:

[0016] Step S31: Deploy the execution script file on the official client server; the execution script file includes the original script file and the calling interface file;

[0017] Step S32: The official client server sends the original script file path to the proxy client server via a parallel job command, and then starts executing the original script file.

[0018] Furthermore, step S3 also includes the proxy client server continuously listening to the received instruction file and executing the script file corresponding to the instruction file, and asynchronously sending back the status information file to the official client server.

[0019] Furthermore, the status information file includes file information that is being executed and file information that has been completed.

[0020] Furthermore, step S5 also includes the scheduling server polling the return value of the original script file executed locally and the return status and result information file of the proxy client server, and sending the execution result of the local machine and the execution result of the proxy client server to the scheduling system.

[0021] Furthermore, the original script file may also include an empty file or a collaborative function script file.

[0022] Furthermore, the official client server is Red Hat, the proxy client server is a Kylin server, and the scheduling system server is a Control M server. This invention also relates to a system for parallel scheduling based on proxy clients, characterized by comprising:

[0023] The module calls the scheduling system server to perform batch jobs by calling the official client server through the client.

[0024] The mapping relationship construction module constructs the mapping relationship between the official client server and the proxy client server. The mapping relationship includes one-to-one mapping, many-to-one mapping, or one-to-many mapping.

[0025] The scheduling module allows the official client server to call the proxy client server to perform batch jobs via parallel job instructions sent by the proxy client.

[0026] The data synchronization module allows the proxy client server to synchronize execution results to the official client server through the proxy client receiving end;

[0027] The result sending module combines the execution results from the official client server and the proxy client server and sends them to the scheduling system server.

[0028] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0029] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0030] The memory is used to store operation instructions;

[0031] The processor is configured to execute the above-described method by invoking the operation instructions.

[0032] A computer program product, comprising a computer program and / or instructions, characterized in that, when the computer program and / or instructions are executed by a processor, they implement the steps of the method described above.

[0033] The beneficial effects of this invention are as follows:

[0034] The parallel scheduling method and system based on a proxy client described in this invention mainly involves the official client server calling the proxy client server through the proxy client sending end to perform batch jobs, and sending the execution results of the local machine and the proxy client server to the scheduling system server. This method uses a proxy client to manage and schedule batch jobs on hosts that cannot have the client installed. It is suitable for situations where the target host cannot install the scheduling management system's client. It allows for the simultaneous scheduling of different controlled hosts through the proxy client, achieving the goal of synchronously scheduling different host jobs in parallel. Compared to developing a new client, this method has lower development costs and a shorter development cycle, and can meet the project's time and schedule requirements in stages. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the method flow for parallel scheduling based on a proxy client according to the present invention.

[0036] Figure 2 This is a schematic diagram of the system architecture of the present invention based on parallel scheduling of agent clients. Detailed Implementation

[0037] In this invention, a batch script refers to a program written using the functions of Linux / Unix / Windows shell / CMD. This program uses a plain text file to write some shell syntax and commands, and then uses regular expressions, pipe commands, and data stream redirection to achieve our desired processing purpose.

[0038] Scheduling and management systems typically adopt a client / server (C / S) model. The management terminal is installed on the server, and the agent client is installed on the controlled host. The client can submit job flows defined by the management terminal on the host and return the running results. The management terminal can comprehensively monitor the batch job running status on all controlled hosts and schedule all controlled hosts to complete the job flows according to the defined host resources, logical resources, and the dependencies of each job.

[0039] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0040] The first aspect of this invention relates to a process flow as follows: Figure 1 The method for parallel scheduling based on agent clients, as shown, includes:

[0041] Step S1: The scheduling system server calls the official client server through the client to perform batch jobs;

[0042] Step S2: Establish the mapping relationship between the official client server and the proxy client server;

[0043] Step S3: The official client server calls the proxy client server to perform batch jobs through the proxy client sending end using parallel job instructions;

[0044] Step S4: The proxy client server synchronizes the execution results to the official client server through the proxy client receiving end;

[0045] Step S5: The official client server combines the execution results from the local machine and the execution results from the proxy client server and sends them to the scheduling system server.

[0046] For example, the mapping relationship between the official client server and the proxy client server includes one-to-one mapping, many-to-one mapping, or one-to-many mapping.

[0047] The proxy client server includes N levels of proxy client servers, where N is greater than or equal to 1.

[0048] The proxy client server includes a receiving end and a sending end, wherein the upper-level proxy client server calls the lower-level proxy client server to perform batch operations through the sending end.

[0049] In another embodiment of the present invention, a method for multi-level parallel scheduling based on a proxy client is disclosed, comprising:

[0050] Step N1: The scheduling system server calls the official client server through the client to perform batch jobs;

[0051] Step N2: Establish the mapping relationship between the official client server and the primary proxy client server;

[0052] Step N3: The official client server calls the first-level agent client server to perform batch jobs through the agent client sending end in the form of parallel job instructions;

[0053] Step N4: The primary agent client server calls the secondary agent client server through the sending end to perform batch operations;

[0054] ...

[0055] Steps N+2 and N-1 are performed by the sending end calling the N-level proxy client server to perform batch operations.

[0056] Step N+3: The proxy client servers at all levels synchronize the execution results to the official client server through the proxy client receiving end;

[0057] Step N+4: The official client server combines the execution results from the local machine and the execution results from the agent client servers at all levels and sends them to the scheduling system server.

[0058] In this embodiment, the number of agent client server levels is configured according to the capacity of the batch job.

[0059] In this embodiment, the lower-level agent client server receives instructions from the upper-level agent client server through the receiving end and returns "received successfully - current CPU memory usage ratio" data to the upper-level agent client server.

[0060] In this invention, the official client server is a Red Hat server, the proxy client server is a Kylin server, and the scheduling system server is a Control M server.

[0061] By way of example, the Red Hat server of this invention includes: servers a1, a2, a3…an;

[0062] Kylin servers include: b servers b1, b2, b3, b4, b5, b6…bn;

[0063] Configure a corresponding Kylin server on each Red Hat server. You can build the following mapping relationships as needed:

[0064] a1 maps to b1, a2 maps to b2, and a3 maps to b3;

[0065] a1 a2 maps to b1, a3 a4 maps to b2, a5 a6 maps to b3;

[0066] a1 maps to b1 b2, a2 ​​maps to b3 b4, and a3 maps to b5 b6;

[0067] a1 maps to b1 b2, a2 ​​maps to b3, a3 maps to b4, and a4 maps to b5 b6;

[0068] The mapping relationship is constructed according to the specific business type. Optionally, step S3 further includes the following sub-steps:

[0069] Step S31: Deploy the execution script file on the official client server; the execution script file includes the original script file and the calling interface file;

[0070] Step S32: The official client server sends the original script file path to the proxy client server via a parallel job command, and then starts executing the original script file.

[0071] For example, the original script file path is the shell path.

[0072] Optionally, step S3 further includes the proxy client server infinitely looping to listen for received instruction files and executing the script files corresponding to the instruction files, and asynchronously sending back status information files to the official client server; wherein the infinite loop listening time interval is set to 500ms.

[0073] Optionally, the status information file includes file information for execution in progress and file information for execution completion. For example, if the script is still executing, it returns "Executing", and if execution has ended, it returns "Execution Complete" and the corresponding execution result.

[0074] Optionally, step S5 further includes the official client server polling the return value of the original script file executed locally and the return status and result information file of the proxy client server, and sending the execution result of the local machine and the execution result of the proxy client server to the scheduling system server.

[0075] Optionally, step S5 further includes the official client server polling and checking the execution status of the proxy client server; if the execution status of the proxy client server is determined to be abnormal, the execution of the proxy client server is terminated.

[0076] Optionally, the original script file may also include an empty file or a co-operation script file.

[0077] In another embodiment of the present invention, the Red Hat server deploys an execution script file; the execution script file is an empty file and a call interface file. The Red Hat server will directly return success when executing the empty file, that is, the Red Hat server does not execute batch jobs, but only starts batch jobs of the Kylin server through a single or multiple Red Hat server agents. The Red Hat server polls the return status and result information file of the Kylin server and sends the execution result of the Kylin server to the scheduling system server.

[0078] In another embodiment of the invention, the Red Hat server deploys the original script files, while the Kylin server deploys the collaborative function script files. That is, the Red Hat server executes batch jobs, while the Kylin server only performs the supporting functions. This method is suitable for scenarios involving real-time data backup in disaster recovery environments.

[0079] In another embodiment of the present invention, the Red Hat server deploys and executes the collaborative function script file, while the Kylin server deploys the original script file. That is, the Red Hat server only executes the associated functions, while the Kylin server executes batch jobs. Specifically, the Red Hat server sends the path to the original script file to the Kylin server via a parallel job command before starting the execution of the collaborative function script file.

[0080] For example, the API call file is: / home / ap / ncismigbat / shell / BATCH_JOB1.sh.

[0081] For example, the execution status file sent by the proxy client server is as follows:

[0082] JOB_STATUS = , where JOB_STATUS is shown in the table below:

[0083] Running In progress Done Execution completed

[0084] JOB_RETURN = , where JOB_RETURN is shown in the table below:

[0085] 0 Execution successful other Execution failed

[0086] The parallel scheduling method and system based on a proxy client described in this invention mainly involves the official client server calling the proxy client server through the proxy client sending end to perform batch jobs, and sending the execution results of the local machine and the proxy client server to the scheduling system server. This method uses a proxy client to manage and schedule batch jobs on hosts that cannot have the client installed. It is suitable for situations where the target host cannot install the scheduling management system's client. It allows for the simultaneous scheduling of different controlled hosts through the proxy client, achieving the goal of synchronously scheduling different host jobs in parallel. Compared to developing a new client, this method has lower development costs and a shorter development cycle, and can meet the project's time and schedule requirements in stages.

[0087] Another aspect of the present invention relates to a system based on parallel scheduling of agent clients, the structure of which is as follows: Figure 2 As shown, it includes:

[0088] The module calls the scheduling system server to perform batch jobs by calling the official client server through the client.

[0089] The mapping relationship construction module constructs the mapping relationship between the official client server and the proxy client server. The mapping relationship includes one-to-one mapping, many-to-one mapping, or one-to-many mapping.

[0090] The scheduling module allows the official client server to call the proxy client server to perform batch jobs via parallel job instructions sent by the proxy client.

[0091] The data synchronization module allows the proxy client server to synchronize execution results to the official client server through the proxy client receiving end;

[0092] The result sending module combines the execution results from the official client server and the proxy client server and sends them to the scheduling system server.

[0093] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0094] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0095] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device includes at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method, such as [examples omitted]. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effect.

[0096] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0097] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0102] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for parallel scheduling based on a proxy client, characterized in that, Includes the following steps: Step S1: The scheduling system server calls the official client server through the client to perform batch jobs; Step S2: Construct a mapping relationship between the official client server and the proxy client server, wherein the mapping relationship includes one-to-one mapping, many-to-one mapping, or one-to-many mapping; Step S3: The official client server calls the proxy client server to perform batch jobs through the proxy client sending end using parallel job instructions; Step S4: The proxy client server synchronizes the execution results to the official client server through the proxy client receiving end; Step S5: The official client server combines the execution results from the local machine and the proxy client server and sends them to the scheduling system server. Step S5 further includes the official client server polling the return value of the original script file executed on the local machine and the return status and result information file of the proxy client server, and sending the execution result of the local machine and the execution result of the proxy client server to the scheduling system server; The method further includes: The original script file may also include an empty file or a co-operation script file; The official client server deploys and executes script files, which are either empty files or API call files. The official client server executes the empty file and returns success directly, meaning that the official client server does not execute batch jobs. Instead, it initiates batch jobs for proxy client servers through one or more official client server proxies. The official client server polls the return status and result information files of the proxy client servers and sends the execution results of the proxy client servers to the scheduling system server. The official client server deploys the original script files, while the proxy client server deploys the collaborative function script files. That is, the official client server executes batch jobs, while the proxy client server only executes the supporting functions. The official client server deploys and executes the collaborative function script file, while the proxy client server deploys the original script file. That is, the official client server only executes the supporting functions, while the proxy client server executes batch jobs. Specifically, the official client server sends the path of the original script file to the proxy client server through parallel job instructions before starting to execute the collaborative function script file.

2. The method as described in claim 1, characterized in that, The proxy client server includes N levels of proxy client servers, where N is greater than or equal to 1.

3. The method as described in claim 2, characterized in that, The proxy client server includes a receiver and a sender. The upper-level proxy client server calls the lower-level proxy client server to perform batch operations through the sender.

4. The method as described in claim 1, characterized in that, Step S3 further includes the following sub-steps: Step S31: Deploy the execution script file on the official client server; the execution script file includes the original script file and the calling interface file; Step S32: The official client server sends the original script file path to the proxy client server via a parallel job command, and then starts executing the original script file.

5. The method as described in claim 4, characterized in that, Step S3 further includes the proxy client server listening to the received instruction file in an infinite loop and executing the script file corresponding to the instruction file, and asynchronously sending back the status information file to the official client server.

6. The method as described in claim 5, characterized in that, The status information file includes file information that is being executed and file information that has been completed.

7. A system based on parallel scheduling of agent clients, characterized in that, include: The module calls the scheduling system server to perform batch jobs by calling the official client server through the client. The mapping relationship construction module constructs the mapping relationship between the official client server and the proxy client server. The mapping relationship includes one-to-one mapping, many-to-one mapping, or one-to-many mapping. The scheduling module allows the official client server to call the proxy client server to perform batch jobs via parallel job instructions sent by the proxy client. The data synchronization module allows the proxy client server to synchronize execution results to the official client server through the proxy client receiving end; The result sending module combines the execution results from the official client server and the execution results from the proxy client server and sends them to the scheduling system server. The result sending module also includes the official client server polling the return value of the original script file executed on the local machine and the return status and result information file of the proxy client server, and sending the local execution result and the proxy client server execution result to the scheduling system server; The system also includes: The original script file may also include an empty file or a co-operation script file; The official client server deploys and executes script files, which are either empty files or API call files. The official client server executes the empty file and returns success directly, meaning that the official client server does not execute batch jobs. Instead, it initiates batch jobs for proxy client servers through one or more official client server proxies. The official client server polls the return status and result information files of the proxy client servers and sends the execution results of the proxy client servers to the scheduling system server. The official client server deploys the original script files, while the proxy client server deploys the collaborative function script files. That is, the official client server executes batch jobs, while the proxy client server only executes the supporting functions. The official client server deploys and executes the collaborative function script file, while the proxy client server deploys the original script file. That is, the official client server only executes the supporting functions, while the proxy client server executes batch jobs. Specifically, the official client server sends the path of the original script file to the proxy client server through parallel job instructions before starting to execute the collaborative function script file.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

9. An electronic device, characterized in that, Including processor and memory; The memory is used to store operation instructions; The processor is configured to execute the method of any one of claims 1 to 6 by invoking the operation instructions.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images