Job timing scheduling method and system, computer device and storage medium

Abstract

Embodiments of the application disclose a job timing scheduling method and system, computer equipment and a storage medium. The method comprises: being applied to a job timing scheduling system, the job timing scheduling system comprising an image management platform, a distributed coordination component, a job coordinator and a K8S cluster, the job configuration of a target job being stored in the distributed coordination component, and the job function of the target job being stored in the image management platform. The method comprises: detecting, by the job coordinator, whether the job execution time in the job configuration of the distributed coordination component has arrived; if the job execution time has arrived, triggering, by the job coordinator, the K8S cluster to start a POD; generating, by the POD, a job process by pulling the job function from the image management platform; executing the target job according to the job process; and recycling, by the job coordinator, the job process after the target job is executed. The resources in the K8S cluster can be scaled as needed, automatic scaling is achieved, and resource waste is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a job timing scheduling method and system, a computer device and a storage medium. BACKGROUND

[0002] With the popularization of Internet technology, in daily work, there are often some needs for batch data processing and analysis, and of course there are also scheduling work according to time period planning.

[0003] A Kubernetes (K8S) cluster is an open source platform for automatic deployment, expansion and operation and maintenance of a container cluster. Through the K8S cluster, user demands can be quickly and effectively responded to, and jobs can be quickly and predictably executed.

[0004] However, in the prior art, a developer needs to manually apply fixed resources for a job in a K8S cluster before the job is executed. After the resources are applied, the applied resources will occupy machine resources before the job is executed and after the job is completed, resulting in serious waste of resources. SUMMARY

[0005] The embodiments of the present application provide a job timing scheduling method and system, a computer device and a storage medium, which can reduce resource waste.

[0006] In a first aspect, the embodiments of the present application provide a job timing scheduling method applied to a job timing scheduling system, the job timing scheduling system comprising an image management platform, a distributed coordination component, a job coordinator and a K8S cluster, the job configuration of a target job being stored in the distributed coordination component, the job function of the target job being stored in the image management platform, and the method comprising:

[0007] detecting, by the job coordinator, whether a job execution time in the job configuration in the distributed coordination component arrives;

[0008] if the job execution time in the job configuration has arrived, triggering, by the job coordinator, the K8S cluster to start a POD;

[0009] generating, by the POD, a job process from the job function pulled from the image management platform;

[0010] executing the target job according to the job process;

[0011] recycling, by the job coordinator, the job process after the target job is executed.

[0012] In a second aspect, the embodiments of the present application further provide a job timing scheduling system, the job timing scheduling system comprising a mirror management platform, a distributed coordination component, a job coordinator and a K8S cluster, wherein:

[0013] The distributed coordination component is configured to store a job configuration of a target job.

[0014] The mirror management platform is configured to store a job function of the target job.

[0015] The job coordinator is configured to detect whether a job execution time in the job configuration in the distributed coordination component has arrived, and trigger the K8S cluster to start a POD when the job execution time in the job configuration has arrived.

[0016] The K8S cluster is configured to generate a job process by pulling the job function from the mirror management platform through the POD, and execute the target job according to the job process.

[0017] The job coordinator is further configured to recycle the job process after the target job is executed.

[0018] In a third aspect, the embodiments of the present application further provide a computer device, which comprises a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0019] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, the computer program comprising program instructions, and the program instructions being capable of implementing the above method when executed by a processor.

[0020] The embodiment of the present application provides a job timing scheduling method, system, computer device and storage medium. The method comprises the following steps: the method is applied to a job timing scheduling system, the job timing scheduling system comprises a mirror image management platform, a distributed coordination component, a job coordinator and a K8S cluster, the job configuration of a target job is stored in the distributed coordination component, and the job function of the target job is stored in the mirror image management platform. The method comprises the following steps: detecting, by the job coordinator, whether the job execution time in the job configuration of the distributed coordination component is reached; if the job execution time in the job configuration is reached, triggering, by the job coordinator, the K8S cluster to start a POD; generating a job process by pulling the job function from the mirror image management platform through the POD; executing the target job according to the job process; and recycling the job process by the job coordinator after the target job is executed. In the present scheme, the K8S cluster is triggered to start the POD when the job needs to be executed, the job process is recycled by the job coordinator after the job is executed, and the POD in the K8S cluster is closed, so that the resources in the K8S cluster can be scaled as needed, automatic expansion and contraction are realized, and resource waste is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0022] Figure 1 The application scenario of the job timing scheduling method provided by the embodiment of the present application is shown in the figure.

[0023] Figure 2 The flowchart of the job timing scheduling method provided by the embodiment of the present application is shown in the figure.

[0024] Figure 3 The sub-flowchart of the job timing scheduling method provided by the embodiment of the present application is shown in the figure.

[0025] Figure 4 The schematic block diagram of the job timing scheduling system provided by the embodiment of the present application is shown in the figure.

[0026] Figure 5 The schematic block diagram of the job timing scheduling system provided by another embodiment of the present application is shown in the figure.

[0027] Figure 6 The schematic block diagram of the computer device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0029] It should be understood that, when used in the specification and the appended claims, the terms "comprise" and "include" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0030] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clearly indicated by the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0031] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0032] The embodiments of the present application provide a job timing scheduling method, system, computer device and storage medium.

[0033] The execution subject of the job timing scheduling method can be a job timing scheduling system provided by the embodiments of the present application, or a computer device integrated with the job timing scheduling system, wherein the job timing scheduling system can be realized in the form of hardware or software, and the computer device can be a terminal or a server, and the terminal can be a smart phone, a tablet computer, a palm computer, or a notebook computer, etc.

[0034] Please refer to Figure 1 , Figure 1 An application scenario of the job timing scheduling method provided by the embodiments of the present application is shown in a schematic diagram. The job timing scheduling method is applied to Figure 1In the job timing scheduling system, the job timing scheduling system includes a building platform, an image management platform, a distributed coordination component, a job coordinator, and a K8S cluster. After a developer develops an original job function according to a business requirement of a target job, the developer submits the original job function and a job configuration corresponding to the target job to the building platform. The building platform performs image processing on the original job function to obtain a job function, and then writes the job function into the image management platform and writes the job configuration into the distributed coordination component. Then, the job coordinator detects whether a job execution time in the job configuration in the distributed coordination component arrives. If the job execution time in the job configuration arrives, the job coordinator triggers the K8S cluster to start a POD according to the job configuration. The POD pulls the job function from the image management platform to generate a job process. The target job is executed according to the job process. After the target job is executed, the job coordinator recycles the job process.

[0035] Figure 2 is a flowchart of a job timing scheduling method provided in the embodiment of the application. As shown in Figure 2 , the method includes the following steps S110-150.

[0036] S110, detecting, by the job coordinator, whether a job execution time in the job configuration in the distributed coordination component arrives.

[0037] The job configuration includes a job execution time (for example, starting to execute at two o'clock in the evening every day), a number of shards (a number of jobs to be split, for example, 10 shards), and a correspondence between the number of shards and the number of PODs (for example, 1:1).

[0038] It should be noted that the job coordinator in the embodiment subscribes to changes of the distributed coordination component. When detecting that there is a new job configuration in the distributed coordination component, the job coordinator synchronizes the job configuration to a job coordination system in the job coordinator. The job coordination system can perform timing job expiration calculation according to the job execution time in the obtained job configuration.

[0039] In the embodiment, Zookeeper is used as the distributed coordination component in the application, and the Zookeeper stores job configuration information, which can ensure that each node of the K8S cluster obtains consistent data and ensures consistency of the configuration in the cluster.

[0040] S120, if the job execution time in the job configuration arrives, triggering, by the job coordinator, the K8S cluster to start a POD.

[0041] If the job coordinator detects that the job execution time in the job configuration has arrived, the state machine in the job coordinator will be used to write the state of the job to be executed in the corresponding job in the distributed coordination component, and then the job coordinator triggers the K8S cluster to start the POD, where the POD is a container component of the K8S cluster.

[0042] Specifically, step S120 includes: determining the number of PODs by the job coordinator according to the number of job shards and the correspondence between the number of job shards and the number of PODs; and triggering the K8S cluster to start the number of PODs by the job coordinator.

[0043] For example, the number of job shards is 10, and the correspondence between the number of job shards and the number of PODs is 1:1, so the number of PODs is equal to the number of job shards, which is also 10, so the job coordinator needs to trigger the K8S cluster to start 10 PODs.

[0044] S130, the POD pulls the job function from the image management platform to generate a job process.

[0045] If there are multiple PODs, each POD needs to pull the job function to generate a corresponding job process.

[0046] In the embodiment, the job function includes business logic, and specifically, each job process generates a job by the business logic, where the business logic includes execution logic of the target job, and in some embodiments, the execution logic includes data processing logic of the target job, for example, receiving information every morning; in other embodiments, the execution logic of the target job includes a data acquisition approach and data processing logic of the target job, for example, the target job is to check bills, and at this time, the data acquisition approach is a bill acquisition path, and the data processing logic is a checking logic.

[0047] Each POD corresponds to a job process, and the job function is started when the POD is started, and the process in each POD detects whether the number of PODs in the current job configuration (the number of PODs in the current job configuration is determined by detecting the number of shards corresponding to the job in the distributed coordination component in the state of being executed) is consistent with the number of currently started PODs, if not, wait for all PODs to be started before executing the job, where each job process writes an IP temporary node in the current job state of the distributed coordination component after being started, and the IP temporary node is invalidated after the process is closed.

[0048] S140, executing the target job according to the job process.

[0049] Referring to Figure 3 In some embodiments, specifically, step S140 comprises:

[0050] S141, detecting the total amount of the target job through the business logic in the job function.

[0051] For example, if the target job is checking bills, in order to slice the job, the total amount of the job to be processed needs to be counted before the job starts.

[0052] In some embodiments, step S141 specifically comprises: determining the job address of the target job in the business logic; and detecting the data amount in the job address to determine the total amount of the job.

[0053] For example, the job address is the address of storing bill data, and the embodiment needs to first preliminarily obtain the total amount of bill data from the address of storing bill data, for example, how many pieces of bill data, and then determine the total amount of bill data as the total amount of the job.

[0054] S142, slicing the total amount of the job according to the number of job slices to obtain the number of slices.

[0055] In some embodiments, after obtaining the total amount of the job, the job will be evenly divided according to the number of job slices, and the total amount of the job will be divided into the number of slices. For example, the total amount of the job is 1 million data, and the number of job slices is 10, so the job can be divided into 10 slices, each slice has 100,000 data.

[0056] S143, respectively allocating the corresponding slice to each job process.

[0057] After obtaining the data amount contained in each slice, each job process will be allocated the data amount of the job, for example, each job process is allocated 100,000 data.

[0058] S144, executing the target job of the corresponding slice according to the job process.

[0059] After each job is allocated the data of the corresponding slice, each job process will read the corresponding slice data and execute the job of the corresponding slice synchronously.

[0060] For example, if the target job is checking bills, after each job process obtains the corresponding slice, the corresponding slice data will be checked in the code.

[0061] In some embodiments, the job coordinator assigns a slice number to each job process according to the number of slices, and then reads corresponding job data according to the slice number to execute the corresponding job according to the job.

[0062] For example, if the data is divided into 10 slices, there are 10 slice numbers, and the slice numbers are 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. Each slice number corresponds to 10 job processes for execution. When the job processes are executed, they can obtain 10 sequence numbers and then split the data according to the sequence numbers. For example, if the job process obtains the sequence number 1, it queries the data related to 1, and if the sequence number is 2, it queries the data related to 2.

[0063] S150, after the target job is executed, the job process is recycled through the job coordinator.

[0064] In this embodiment, the target job is not always being executed. In order to avoid idle processes occupying machine resources all the time, the job process will be recycled after the target job is executed.

[0065] Specifically, step S150 includes: after the job process executes the target job of the corresponding slice, writing a completion mark of the corresponding slice in the distributed coordination component; and when the job coordinator detects that all job processes have written the completion mark, recycling the job process.

[0066] That is, after each job process completes the job of the corresponding slice, it writes a completion mark of the corresponding slice in the distributed coordination component. Since the job coordinator subscribes to changes in the data in the distributed coordination component, the job coordinator can detect whether all job processes have written the job completion mark. When all job processes have written the completion mark, it means that the target job has been completed, and the job process is recycled.

[0067] In some embodiments, during job execution, the job coordinator can periodically pull resource usage data of each job process, such as CPU, memory, etc., and then generate a resource adjustment scheme according to the obtained resource usage data. For example, if the CPU usage is high, a resource adjustment scheme to increase server resources is generated. If the CPU usage is low, a resource adjustment scheme to reduce server resources is generated.

[0068] After generating the resource adjustment scheme, the server resources can be adjusted manually or automatically according to the obtained resource adjustment scheme. After the job process is recycled by the job coordinator, the method further includes adjusting the server resources corresponding to the K8S cluster according to the resource adjustment scheme.

[0069] It can be seen that the server resources in the embodiment can be scaled as needed, so that the server resources are more reasonably utilized, resource waste caused by resource excess is avoided, and low execution efficiency caused by resource deficiency is avoided.

[0070] In some embodiments, the job timing scheduling system further comprises a build platform, before the job execution time in the job configuration in the distributed coordination component is detected by the job coordinator, the method further comprises: receiving, by the build platform, an original job function of the target job and the job configuration; performing, by the build platform, mirror processing on the original job function to obtain the job function; writing the job function into the image management platform and writing the job configuration into the distributed coordination component.

[0071] The job timing scheduling system further comprises a job development function jar component, the job timing scheduling system can automatically develop an original job function according to a business requirement of a user in the job development function jar component and obtain a job configuration input by the user, then the build platform packs the original job function and a preset job framework to build a complete project package, builds the project package into an image (job function), uploads the image to an image management platform, and the build platform writes the job configuration into the distributed coordination component.

[0072] In the embodiment, the developer does not need to consider the underlying technical implementation, can fully exert the elasticity and distributed advantage of the cloud platform, realizes rapid deployment, on-demand scaling and rapid delivery, and only needs to focus on the development of the core job business logic, can only care about the core business to develop the job, automatically publishes and deploys the system, and can quickly adjust the number of shards and POD parameters according to the business data and collected index data to realize horizontal expansion, guarantee resource full utilization, when the job is not in the running time or is executed, the container resources are all recycled by the job coordinator, and it is guaranteed that the idle resources will not be wasted.

[0073] To sum up, the scheme is applied to a job timing scheduling system, the job timing scheduling system comprises a mirror management platform, a distributed coordination component, a job coordinator and a K8S cluster, the job configuration of a target job is stored in the distributed coordination component, the job function of the target job is stored in the mirror management platform, and the method comprises the following steps: detecting, by the job coordinator, whether the job execution time in the job configuration in the distributed coordination component arrives; if the job execution time in the job configuration arrives, triggering, by the job coordinator, the K8S cluster to start a POD; generating, by the POD, a job process by pulling the job function from the mirror management platform; executing the target job according to the job process; and recycling, by the job coordinator, the job process after the target job is executed. In the scheme, the K8S cluster is triggered to start a POD when a job needs to be executed, the job process is recycled by the job coordinator after the job is executed, and the POD in the K8S cluster is closed, so that the resources in the K8S cluster can be scaled as needed, automatic scaling is realized, and resource waste is reduced.

[0074] Figure 4 is a schematic block diagram of a job timing scheduling system provided by an embodiment of the present application. As shown in Figure 4 Corresponding to the above job timing scheduling method, the present application also provides a job timing scheduling system. The job timing scheduling system comprises units for executing the above job timing scheduling method, and the device can be configured in a desktop computer, a tablet computer, a laptop computer, and the like. Specifically, the job timing scheduling system comprises a mirror management platform 401, a distributed coordination component 402, a job coordinator 403 and a K8S cluster 404, wherein:

[0075] The distributed coordination component 402 is configured to store the job configuration of a target job.

[0076] The mirror management platform 401 is configured to store the job function of a target job,

[0077] The job coordinator 403 is configured to detect whether the job execution time in the job configuration in the distributed coordination component 402 arrives; when the job execution time in the job configuration arrives, the K8S cluster 404 is triggered to start a POD.

[0078] The K8S cluster 404 is configured to generate a job process by pulling the job function from the mirror management platform 401 by the POD; and execute the target job according to the job process.

[0079] The job coordinator 403 is further configured to recycle the job process after the target job is executed.

[0080] In some embodiments, the job configuration further comprises a number of job shards and a correspondence between the number of job shards and the number of PODs, and the job coordinator 403 is specifically configured to:

[0081] determine the number of PODs according to the number of job shards and the correspondence between the number of job shards and the number of PODs; and trigger the K8S cluster 404 to start the number of PODs.

[0082] In some embodiments, the K8S cluster 404 is specifically configured to: respectively pull the job function from the image management platform 401 by each of the number of PODs to generate the number of job processes.

[0083] In some embodiments, the K8S cluster 404, when executing the target job according to the job process, is further specifically configured to:

[0084] detect a total amount of the target job by the business logic in the job function; slice the total amount of the target job according to the number of job shards to obtain a number of slices; respectively allocate a corresponding slice to each of the job processes; and execute the target job of the corresponding slice according to the job process.

[0085] In some embodiments, the K8S cluster 404, when detecting the total amount of the target job by the business logic in the job function, is further specifically configured to:

[0086] determine a job address of the target job in the business logic; and detect an amount of data in the job address to determine the total amount of the target job.

[0087] In some embodiments, the K8S cluster 404 is further configured to write a completion marker of the corresponding slice in the distributed coordination component after the job process executes the target job of the corresponding slice.

[0088] The job coordinator 403 is further configured to recycle the job process after detecting that all job processes have written the completion marker.

[0089] In some embodiments, the job coordinator 403 is further configured to read resource usage data of the job process; generate a resource adjustment scheme according to the resource usage data; and adjust server resources corresponding to the K8S cluster according to the resource adjustment scheme.

[0090] Figure 5 is a schematic block diagram of a job timing scheduling system according to another embodiment of the present application. As shown in Figure 5As shown, the job timing scheduling system of the embodiment is based on the above-mentioned embodiment and increases the build platform 405.

[0091] The build platform 405 is configured to receive the original job function of the target job and the job configuration, perform mirror processing on the original job function to obtain the job function, write the job function into the mirror management platform, and write the job configuration into the distributed coordination component.

[0092] It should be noted that the specific implementation process of the above-mentioned job timing scheduling system and each unit can be clearly understood by those skilled in the art, and the corresponding description in the foregoing method embodiment can be referred to. For the convenience and brevity of description, it will not be repeated here.

[0093] The above-mentioned job timing scheduling system can be implemented in the form of a computer program, which can run on a computer device as shown. Figure 6

[0094] Please refer to Figure 6 , Figure 6 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 600 can be a terminal or a server, which integrates a job timing scheduling system. The job timing scheduling system includes a mirror management platform, a distributed coordination component, a job coordinator, and a K8S cluster. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, a wearable device, and other electronic devices with communication functions. The server can be a standalone server or a server cluster composed of multiple servers.

[0095] Please refer to Figure 6 , the computer device 600 includes a processor 602, a memory, and a network interface 605 connected through a system bus 601. The memory can include a non-volatile storage medium 603 and an internal memory 604.

[0096] The non-volatile storage medium 603 can store an operating system 6031 and a computer program 6032. The computer program 6032 includes program instructions which, when executed, can cause the processor 602 to perform a job timing scheduling method.

[0097] The processor 602 is configured to provide computing and control capabilities to support the operation of the entire computer device 600.

[0098] The internal memory 604 provides an environment for the operation of the computer program 6032 in the non-volatile storage medium 603. When the computer program 6032 is executed by the processor 602, the processor 602 can perform a job timing scheduling method.​

[0099] The network interface 605 is configured to perform network communication with other devices. Those skilled in the art can understand that the network interface 605 can be implemented by a network card such as an Ethernet card or a modem (modulator-demodulator), etc. Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 600 to which the scheme of the present application is applied. Specifically, the computer device 600 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] The distributed coordination component stores a job configuration of a target job, and the image management platform stores a job function of the target job. The processor 602 is configured to run the computer program 6032 stored in the memory to implement the following steps:

[0101] Detecting, by the job coordinator, whether a job execution time in the job configuration in the distributed coordination component arrives;

[0102] If the job execution time in the job configuration arrives, triggering, by the job coordinator, the K8S cluster to start a POD;

[0103] Pulling, by the POD, the job function from the image management platform to generate a job process;

[0104] Executing the target job according to the job process;

[0105] After the target job is executed, recycling, by the job coordinator, the job process.

[0106] In some embodiments, the job configuration further includes a job shard number and a corresponding relationship between the job shard number and a POD number. When implementing the step of triggering, by the job coordinator, the K8S cluster to start a POD, the processor 602 specifically implements the following steps:

[0107] Determining, by the job coordinator, a number of PODs according to the job shard number and the corresponding relationship between the job shard number and the POD number;

[0108] Triggering, by the job coordinator, the K8S cluster to start the number of PODs;

[0109] The step of pulling, by the POD, the job function from the image management platform to generate a job process includes:

[0110] Pulling, by each of the number of PODs, the job function from the image management platform respectively to generate the number of job processes.

[0111] In some embodiments, the processor 602, in implementing the step of executing the target job according to the job process, implements the following steps:

[0112] detecting a total amount of the target job through business logic in the job function;

[0113] slicing the total amount of the target job according to the number of job shards to obtain a slice of the number of job shards;

[0114] allocating the corresponding slice to each of the job processes respectively;

[0115] executing the target job of the corresponding slice according to the job process.

[0116] In some embodiments, the processor 602, in implementing the step of detecting the total amount of the target job through business logic in the job function, implements the following steps:

[0117] determining a job address of the target job in the business logic;

[0118] detecting an amount of data in the job address to determine the total amount of the target job.

[0119] In some embodiments, after the processor 602 implements the step of recycling the job processes through the job coordinator after the target job is executed, the processor 602 further implements the following steps:

[0120] writing a completion marker of the corresponding slice in the distributed coordination component after the job process executes the target job of the corresponding slice;

[0121] recycling the job processes after the job coordinator detects that all job processes have written the completion marker.

[0122] In some embodiments, after the processor 602 implements the step of executing the target job according to the job process, the processor 602 further implements the following steps:

[0123] reading resource usage data of the job process through the job coordinator;

[0124] generating a resource adjustment scheme according to the resource usage data;

[0125] adjusting server resources corresponding to the K8S cluster according to the resource adjustment scheme.

[0126] In some embodiments, the job timing scheduling system further comprises a construction platform, and before the processor 602 implements the step of detecting, through the job coordinator, whether the job execution time in the job configuration in the distributed coordination component has arrived, the processor 602 further implements the following steps:

[0127] receive, by the build platform, an original job function of the target job and the job configuration;

[0128] mirror, by the build platform, the original job function to obtain the job function;

[0129] write the job function into the image management platform and write the job configuration into the distributed coordination component.

[0130] It should be understood that, in the embodiments of the present application, the processor 602 can be a central processing unit (CPU), and the processor 602 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0131] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments.

[0132] Therefore, the present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions are executed by the processor to make the processor perform the following steps:

[0133] detect, by the job coordinator, whether the job execution time in the job configuration in the distributed coordination component has arrived;

[0134] if the job execution time in the job configuration has arrived, trigger, by the job coordinator, the K8S cluster to start a POD;

[0135] generate, by the POD, a job process from the job function pulled from the image management platform;

[0136] executing the target job according to the job process;

[0137] reclaiming the job process through the job coordinator after the target job is executed.

[0138] In some embodiments, the job configuration further comprises a number of job shards and a correspondence between the number of job shards and the number of PODs, and when the processor executes the program instructions to implement the step of triggering the K8S cluster to start the PODs through the job coordinator, the following steps are implemented:

[0139] determining the number of PODs through the job coordinator according to the number of job shards and the correspondence between the number of job shards and the number of PODs;

[0140] triggering the K8S cluster to start the number of PODs through the job coordinator;

[0141] the step of pulling the job function generation job process from the image management platform through the POD, comprising:

[0142] pulling the job function from the image management platform through each of the number of PODs to generate the number of job processes.

[0143] In some embodiments, when the processor executes the program instructions to implement the step of executing the target job according to the job process, the following steps are implemented:

[0144] detecting the total amount of the target job through the business logic in the job function;

[0145] slicing the total amount of the target job according to the number of job shards to obtain a number of slices;

[0146] allocating a corresponding slice to each of the job processes;

[0147] executing the target job of the corresponding slice according to the job process.

[0148] In some embodiments, when the processor executes the program instructions to implement the step, the following steps are implemented: 4. The method of claim 3, wherein the step of detecting the total amount of the target job through the business logic in the job function comprises:

[0149] determining the job address of the target job in the business logic;

[0150] detecting the amount of data in the job address to determine the total amount of the target job.

[0151] In some embodiments, the processor, after executing the program instructions to implement the step of recycling the job processes through the job coordinator after the target job is executed, further implements the following steps:

[0152] writing a completion flag corresponding to the slice in the distributed coordination component after the job process executes the target job corresponding to the slice;

[0153] recycling the job processes after the job coordinator detects that all job processes have written the completion flag.

[0154] In some embodiments, the processor, after executing the program instructions to implement the step of executing the target job according to the job processes, further implements the following steps:

[0155] reading resource usage data of the job processes through the job coordinator;

[0156] generating a resource adjustment scheme according to the resource usage data;

[0157] adjusting server resources corresponding to the K8S cluster according to the resource adjustment scheme.

[0158] In some embodiments, the job timing scheduling system further comprises a construction platform, and the processor, before executing the program instructions to implement the step of detecting, through the job coordinator, whether the job execution time in the job configuration in the distributed coordination component arrives, further implements the following steps:

[0159] receiving, through the construction platform, an original job function of the target job and the job configuration;

[0160] performing mirror processing on the original job function through the construction platform to obtain the job function;

[0161] writing the job function in the image management platform and writing the job configuration in the distributed coordination component.

[0162] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.

[0163] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0164] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed.

[0165] The steps in the method embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the apparatus embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0166] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions to make a computer device (which can be a personal computer, a terminal or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.

[0167] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A job timing scheduling method characterized by comprising: The application is applied to a job timing scheduling system, the job timing scheduling system comprises a mirror management platform, a distributed coordination component, a job coordinator and a K8S cluster, the job configuration of a target job is stored in the distributed coordination component, the job function of the target job is stored in the mirror management platform, the job function comprises business logic, the business logic comprises execution logic of the target job or comprises data acquisition approach and data processing logic of the target job, and the method comprises the following steps: whether the job execution time in the job configuration in the distributed coordination component reaches through the job coordinator; if the job execution time in the job configuration reaches, the K8S cluster is triggered to start a POD through the job coordinator; the job function is pulled from the mirror management platform through the POD to generate a job process; the target job is executed according to the job process; after the target job is executed, the job process is recycled through the job coordinator; the job configuration further comprises the number of job fragments and the corresponding relationship between the number of job fragments and the number of PODs, and the K8S cluster is triggered to start a POD through the job coordinator, which comprises the following steps: the number of PODs is determined through the job coordinator according to the number of job fragments and the corresponding relationship between the number of job fragments and the number of PODs; the K8S cluster is triggered to start the number of PODs through the job coordinator; the job function is pulled from the mirror management platform through the POD to generate a job process, which comprises the following steps: the job function is pulled from the mirror management platform through each of the number of PODs to generate the number of job processes.

2. The method of claim 1, wherein, the target job is executed according to the job process, which comprises the following steps: the total amount of the target job is detected through the business logic in the job function; the total amount of the target job is sliced according to the number of job fragments to obtain the number of slices of the target job; each job process is allocated a corresponding slice; the target job of the corresponding slice is executed according to the job process.

3. The method of claim 2, wherein, the total amount of the target job is detected through the business logic in the job function, which comprises the following steps: the job address of the target job in the business logic is determined; the amount of data in the job address is detected to determine the total amount of the target job.

4. The method of claim 2, wherein, after the target job is executed, the job process is recycled through the job coordinator, which comprises the following steps: after the job process executes the target job of the corresponding slice, a completion mark of the corresponding slice is written in the distributed coordination component; after the job coordinator detects that all job processes have written the completion mark, the job process is recycled.

5. The method of claim 1, wherein, after the target job is executed according to the job process, the method further comprises the following steps: the resource usage data of the job process is read through the job coordinator; a resource adjustment scheme is generated according to the resource usage data; the server resources corresponding to the K8S cluster are adjusted according to the resource adjustment scheme.

6. The method according to any one of claims 1 to 5, characterized in that, The job timing scheduling system further comprises a construction platform, and before the job execution time in the job configuration in the distributed coordination component is detected by the job coordinator, the method further comprises: An original job function of the target job is received by the construction platform, and the job configuration is received by the construction platform; The original job function is mirrored by the construction platform to obtain the job function; The job function is written into the image management platform, and the job configuration is written into the distributed coordination component.

7. A job timing scheduling system characterized by comprising: The job timing scheduling system comprises an image management platform, a distributed coordination component, a job coordinator and a K8S cluster, wherein: The distributed coordination component is configured to store a job configuration of a target job; The image management platform is configured to store a job function of the target job, the job function comprising business logic, the business logic comprising execution logic of the target job or comprising data acquisition approach and data processing logic of the target job; The job coordinator is configured to detect whether a job execution time in the job configuration in the distributed coordination component has arrived, and when the job execution time in the job configuration has arrived, trigger the K8S cluster to start a POD; The K8S cluster is configured to generate a job process by the POD pulling the job function from the image management platform, and execute the target job according to the job process; The job coordinator is further configured to recycle the job process after the target job is executed; The job configuration further comprises a job shard quantity and a corresponding relationship between the job shard quantity and a POD quantity, and the job coordinator is specifically configured to determine a POD quantity according to the job shard quantity and the corresponding relationship between the job shard quantity and the POD quantity, and trigger the K8S cluster to start the POD quantity of PODs; The K8S cluster is specifically configured to pull the job function from the image management platform by each of the POD quantity of PODs to generate the job process of the POD quantity.

8. A computer device, comprising: The computer device integrates a job timing scheduling system, the computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, the computer program comprises program instructions, and the program instructions can realize the method in any one of claims 1-6 when executed by a processor.

Images