Job processing method and device, equipment and storage medium
By implementing file arrival detection and unified file detection interfaces in the batch executor of the distributed batch platform, errors and batch response interrupts caused by the failure of dependency files are solved, and the efficiency of job processing and the overall processing capability of the platform are improved.
Patent Information
- Application Number
- CN202411923592.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-04-01
AI Technical Summary
In the distributed batch platform, when batch executors process jobs, they fail to consider whether the job dependency files arrive, which affects processing efficiency. Moreover, the file detection methods of each batch executor are not uniform, resulting in batch response interruption.
By implementing file arrival detection in batch executors, using the file detection interface uniformly configured by the distributed batch platform, the dependent files of the target job are detected, and the job processing is only performed after all dependent files arrive, ensuring the accuracy and efficiency of processing.
It effectively avoids errors caused by the failure of dependency files, improves the processing efficiency of batch executors for jobs, and avoids batch response interrupts through a unified file detection interface, and improves the processing efficiency of batch jobs by distributed batch platforms.
Smart Images
Figure CN119739484A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distributed technology, and in particular to a job processing method, device, equipment and storage medium. Background Art
[0002] With the continuous development and innovation of Internet financial services, distributed batch platforms have been widely used in financial institutions represented by banks and play an important role. However, when the batch executor in the distributed batch platform processes jobs according to the received job scheduling instructions, it often reports errors because it does not consider whether the dependent files on which the job depends have arrived, which affects the processing efficiency of the batch executor for the job.
[0003] In order to solve the above-mentioned problem while avoiding batch response interruption caused by the inconsistency of file detection methods of each batch executor in the distributed batch platform, the present invention provides a solution. Summary of the invention
[0004] The present invention provides a job processing method, device, equipment and storage medium to improve the processing efficiency of the batch executor for the target job while solving the batch response interruption problem caused by the inconsistent file detection method of each batch executor in the distributed batch platform, thereby improving the processing efficiency of the distributed batch platform for batch jobs.
[0005] According to one aspect of the present invention, a job processing method is provided, the method being executed by a batch executor, comprising:
[0006] After receiving the target job scheduling instruction sent by the distributed batch controller, the target job scheduling instruction is parsed to obtain instruction parsing content; wherein the instruction parsing content includes the target job identifier, target processing mode and file dependency of the target job;
[0007] Determine at least one dependent file corresponding to the target job according to the file dependency relationship;
[0008] Perform file arrival detection on each dependent file through the file detection interface; wherein the file detection interface is uniformly configured by the distributed batch platform;
[0009] When it is detected that all dependent files have arrived, the target job is processed using the target processing method.
[0010] According to another aspect of the present invention, there is provided a job processing device, which is applied to a batch executor and includes:
[0011] The instruction parsing content determination module is used to parse the target job scheduling instruction after receiving the target job scheduling instruction sent by the distributed batch controller to obtain the instruction parsing content; wherein the instruction parsing content includes the target job identification, target processing mode and file dependency of the target job;
[0012] A dependent file determination module, used to determine at least one dependent file corresponding to the target job according to the file dependency relationship;
[0013] A file arrival detection module is used to perform file arrival detection on each dependent file through a file detection interface; wherein the file detection interface is uniformly configured by a distributed batch platform;
[0014] The job processing module is used to process the target job in a target processing manner when it is detected that all dependent files have arrived.
[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0016] at least one processor; and
[0017] a memory communicatively connected to at least one processor; wherein,
[0018] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the job processing method of any embodiment of the present invention.
[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the job processing method of any embodiment of the present invention when executed.
[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the job processing method of any embodiment of the present invention.
[0021] The technical solution of the embodiment of the present invention, after receiving the target job scheduling instruction sent by the distributed batch controller, parses the target job scheduling instruction to obtain the instruction parsing content; wherein the instruction parsing content includes the target job identification, target processing method and file dependency of the target job; according to the file dependency, at least one dependent file corresponding to the target job is determined; file arrival detection is performed on each dependent file through the file detection interface; wherein the file detection interface is uniformly configured by the distributed batch platform; when it is detected that each dependent file has arrived, the target processing method is used to process the target job. In the above technical solution, after the batch executor parses the target job scheduling instruction to obtain the instruction parsing content, it determines at least one dependent file corresponding to the target job according to the file dependency, and when it is detected that each dependent file corresponding to the target job has arrived, it processes the target job in the target processing method, thereby avoiding errors caused by the non-arrival of the dependent files corresponding to the target job during the batch executor's processing of the target job, thereby improving the processing efficiency of the batch executor for the target job. At the same time, the batch executor uses the file detection interface uniformly configured by the distributed batch platform to detect whether the dependent files corresponding to the target job have arrived, avoiding batch response interruptions caused by inconsistent file detection methods of each batch executor in the distributed batch platform, thereby improving the processing efficiency of the distributed batch platform for batch jobs.
[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0024] Figure 1 is a flowchart of a job processing method provided according to Embodiment 1 of the present invention;
[0025] Figure 2 is a flowchart of a job processing method provided according to Embodiment 2 of the present invention;
[0026] Figure 3 is a structural schematic diagram of a job processing device provided according to Embodiment 3 of the present invention;
[0027] Figure 4It is a schematic diagram of the structure of an electronic device for implementing the job processing method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0029] It should be noted that the term "target" and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] Embodiment 1
[0031] Figure 1 This is a flowchart of a job processing method provided in the first embodiment of the present invention. This embodiment can be applied to the situation where a batch executor in a distributed batch platform processes jobs according to received job scheduling instructions. The method can be executed by a job processing device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device with a batch executor. It should be noted that a batch executor refers to an application node in a distributed batch platform that is used to execute specific job processing tasks according to job scheduling instructions. Figure 1 As shown, the method is executed by a batch executor and includes:
[0032] S101. After receiving a target job scheduling instruction sent by a distributed batch controller, the target job scheduling instruction is parsed to obtain instruction parsing content; wherein the instruction parsing content includes a target job identifier, a target processing method, and a file dependency of the target job.
[0033] Among them, the distributed batch controller refers to the intelligent equipment in the distributed batch platform that is responsible for job scheduling, job execution process control and resource management. The distributed batch platform refers to a platform that processes and manages large-scale batch jobs based on distributed computing technology; optionally, the distributed batch platform includes a distributed batch controller, an operation and maintenance console, a registration center, a database and at least one batch executor. Among them, the operation and maintenance console is used to show the job scheduling status and job execution progress to the operation and maintenance personnel; the registration center is used to provide registration services for the distributed batch controller and batch executor, and is used to transmit batch task messages and configure the task parameter information of batch tasks; the database is used to store job metadata and file metadata in the distributed batch platform, as well as job execution status.
[0034] Among them, the target job scheduling instruction refers to the job scheduling instruction sent by the distributed batch controller to the batch executor. The target job refers to the job specified in the target job scheduling instruction. The target job identifier refers to the information used to uniquely identify the target job; optionally, the target job identifier can be in the form of numbers, letters, or numbers and letters. The target processing method refers to the method used to process the target job. The file dependency refers to the dependency between the target job and the files involved. The instruction parsing content refers to the content obtained after the target job instruction is parsed; optionally, the instruction parsing content can also include parameters and configuration information required for job execution.
[0035] Specifically, after receiving the target job scheduling instruction sent by the distributed batch controller, the batch executor may parse the target job scheduling instruction according to the encoding format of the target job scheduling instruction to obtain the instruction parsing content.
[0036] S102: Determine at least one dependent file corresponding to the target job according to the file dependency relationship.
[0037] The dependent file refers to a file that has a file dependency relationship with the target job; optionally, the dependent file may be a data file, a configuration file, or a script file.
[0038] Specifically, the batch executor may determine at least one dependent file corresponding to the target job according to the file dependency relationship, so as to ensure the accuracy of the dependent file corresponding to the target job.
[0039] S103: Perform file arrival detection on each dependent file through a file detection interface.
[0040] Among them, the file detection interface is uniformly configured by the distributed batch platform and is used to perform file arrival detection on each dependent file corresponding to the target job.
[0041] Specifically, for each dependent file corresponding to the target job, the batch executor can send the dependent file to the idle detection engine through the file detection interface, and the idle detection engine performs file arrival detection on the dependent file. The idle detection engine refers to the detection engine with idle resources associated with the file detection interface.
[0042] S104: When it is detected that all dependent files have arrived, the target job is processed in a target processing manner.
[0043] Specifically, when the batch executor detects that all dependent files corresponding to the target job have reached the designated storage location, it processes the target job in a target processing manner.
[0044] Optionally, before the batch executor processes the target job in the target processing mode, it can also detect whether the job execution thread pool is full; if so, the target job is added to the queue of jobs to be executed; otherwise, the target job is processed in the target processing mode.
[0045] Optionally, when the batch executor detects that at least one dependent file among the dependent files has not arrived, the batch executor stores the target job in a job waiting queue to wait for the unreached dependent file to arrive at its designated storage location.
[0046] The technical solution of the embodiment of the present invention, after receiving the target job scheduling instruction sent by the distributed batch controller, parses the target job scheduling instruction to obtain the instruction parsing content; wherein the instruction parsing content includes the target job identification, target processing method and file dependency of the target job; according to the file dependency, at least one dependent file corresponding to the target job is determined; file arrival detection is performed on each dependent file through the file detection interface; when it is detected that each dependent file has arrived, the target job is processed in the target processing method. In the above technical solution, after the batch executor parses the target job scheduling instruction to obtain the instruction parsing content, it determines at least one dependent file corresponding to the target job according to the file dependency, and when it is detected that each dependent file corresponding to the target job has arrived, it processes the target job in the target processing method, thereby avoiding errors caused by the non-arrival of the dependent files corresponding to the target job during the batch executor's processing of the target job, thereby improving the processing efficiency of the batch executor for the target job. At the same time, the batch executor uses the file detection interface uniformly configured by the distributed batch platform to detect whether the dependent files corresponding to the target job have arrived, avoiding batch response interruptions caused by inconsistent file detection methods of each batch executor in the distributed batch platform, thereby improving the processing efficiency of the distributed batch platform for batch jobs.
[0047] On the basis of the above-mentioned embodiment, as an optional method of the embodiment of the present invention, the batch executor can also feed back the target processing result of the target job to the distributed batch controller, so that the distributed batch controller can control whether to schedule the subsequent jobs of the target job according to the target processing result and the dependency relationship between the previous and next jobs.
[0048] The target processing result refers to the processing result of the target job by the batch executor. The job pre- and post-job dependencies refer to the relationship between the execution order of batch jobs in the distributed batch platform. Optionally, the job pre- and post-job dependencies can be viewed through the operation and maintenance console in the distributed batch platform.
[0049] Among them, the distributed batch controller controls whether to schedule the subsequent jobs of the target job according to the target processing result and the dependency relationship between the previous and next jobs. Specifically, it can be: the distributed batch controller uses the job identifier of the target job as an index to determine the subsequent jobs of the target job from the dependency relationship between the previous and next jobs; then, according to the target processing result, it controls whether to schedule the subsequent jobs of the target job, that is, if the target processing result is a successful processing, the distributed batch controller generates a job scheduling instruction for the subsequent jobs of the target job; if the target processing result is a failed processing, the distributed batch controller delays the job scheduling of the subsequent jobs of the target job.
[0050] For example, it is known that the target job is job A, and the job dependencies are: job A→job B, job A→job C, job B→job D, job C→job D. The distributed batch controller uses the job identifier of the target job as an index to determine the subsequent jobs of the two target jobs, namely, job B and job C, from the above job dependencies. If the target processing result of the batch executor for job A is successful processing, the distributed batch controller generates job scheduling instructions for job B and job C respectively. If the target processing result of the batch executor for job A is a processing failure, the distributed batch controller delays the job scheduling of job B and job C.
[0051] It can be understood that the distributed batch controller controls whether to schedule the subsequent jobs of the target job based on the target processing results and the dependencies between the previous and next jobs, which helps the distributed batch controller to arrange the scheduling order of batch jobs more reasonably, thereby improving the processing efficiency of batch jobs on the distributed batch platform.
[0052] Embodiment 2
[0053] Figure 2This is a flowchart of a job processing method provided in Example 2 of the present invention. Based on the above example, this example further optimizes "performing file arrival detection on each dependent file through a file detection interface" and provides an optional implementation scheme. It should be noted that for the parts not described in detail in the example of the present invention, reference can be made to the relevant descriptions of other examples. Figure 2 As shown, the method is executed by a batch executor and includes:
[0054] S201. After receiving the target job scheduling instruction sent by the distributed batch controller, the target job scheduling instruction is parsed to obtain instruction parsing content; wherein the instruction parsing content includes the target job identifier, target processing method and file dependency of the target job.
[0055] S202: Determine at least one dependent file corresponding to the target job according to the file dependency relationship.
[0056] S203: For each dependent file, according to the file identifier of the dependent file, obtain the designated arrival location of the dependent file from the file metadata in the distributed batch platform.
[0057] The file identifier of the dependent file refers to information used to uniquely identify the dependent file; optionally, the file identifier of the dependent file may be a file name, a file path, or a file hash value. The specified destination location refers to a location that the dependent file needs to reach that is specified in advance.
[0058] Specifically, for each dependent file corresponding to the target job, the batch executor uses the file identifier of the dependent file as an index and obtains the specified arrival location of the dependent file from the file metadata of the distributed platform.
[0059] Optionally, the file metadata in the distributed batch platform is uniformly defined and published by the metadata platform to ensure that all batch executors have consistent understanding and use of the file metadata during the distributed batch job scheduling process, avoiding errors or confusion caused by inconsistent file metadata definitions, thereby achieving unified management of file metadata in the distributed batch platform. Optionally, the file metadata in the distributed batch platform includes but is not limited to file names, file identifiers, file wildcards, file parsing rules, and specified arrival locations.
[0060] S204: Detect, through a file detection interface, whether the dependent file has reached a specified location.
[0061] Specifically, the batch executor detects whether the dependent file exists at the specified arrival position of the dependent file through the file detection interface.
[0062] S205: If yes, determine that the dependent file has been reached.
[0063] Specifically, if the batch executor detects through the file detection interface that the dependent file has reached the specified arrival position, that is, if the batch executor detects through the file detection interface that the dependent file exists at the specified arrival position of the dependent file, it is determined that the dependent file has been reached.
[0064] S206: If not, it is determined that the dependent file has not been reached.
[0065] Specifically, if the batch executor detects through the file detection interface that the dependent file has not reached the specified arrival position, that is, if the batch executor detects through the file detection interface that the dependent file does not exist at the specified arrival position of the dependent file, it is determined that the dependent file has not been reached.
[0066] Optionally, after determining that the dependent file has not been reached, the batch executor stores the dependent file in a file detection queue and records the file waiting time; based on the file identifier of the dependent file, the batch executor obtains the specified waiting time and the specified alarm time corresponding to the dependent file from the file metadata in the distributed batch platform; when it is detected that the file waiting time exceeds the specified waiting time, the alarm mechanism is triggered, an alarm message is sent to the operation and maintenance personnel, and the file alarm time is recorded; when it is detected that the file alarm time exceeds the specified alarm time, the processing of the target job is forcibly terminated.
[0067] Among them, the file detection queue refers to a queue for storing dependent files that have not arrived. The file waiting time refers to the time waiting for the dependent file to reach its designated arrival location. The prescribed waiting duration refers to a waiting duration set in advance; optionally, the prescribed waiting duration can be pre-set based on the experience of those skilled in the art, and the embodiments of the present invention do not make specific limitations on it. The prescribed alarm duration refers to a prescribed alarm duration; optionally, the prescribed alarm duration can be pre-set based on the experience of those skilled in the art, and the embodiments of the present invention do not make specific limitations on it. The file alarm duration refers to the duration of the alarm after the dependent file triggers the alarm mechanism.
[0068] Specifically, after determining that the dependent file has not been reached, the batch executor stores the dependent file in the file detection queue and records the file waiting time; the batch executor uses the file identifier of the dependent file as an index to obtain the specified waiting time and the specified alarm time corresponding to the dependent file from the file metadata in the distributed batch platform; when the batch executor detects that the file waiting time exceeds the specified waiting time, it triggers the alarm mechanism and sends an alarm message to the operation and maintenance personnel to remind the operation and maintenance personnel to check and correct the dependent file in time, and record the file alarm time; when the batch executor detects that the file alarm time exceeds the specified alarm time, the operation and maintenance personnel forcibly terminate the processing of the target job to prevent the target job from continuously occupying the computing resources and memory resources of the batch executor, avoid job backlogs, and ensure the normal operation of other jobs in the batch executor.
[0069] Optionally, in order to avoid job backlog, the batch executor can also periodically perform file arrival detection on dependent files in the file detection queue.
[0070] Specifically, the batch executor may periodically perform file arrival detection on dependent files in the file detection queue according to a preset file detection frequency.
[0071] Optionally, in order to make the batch executor's timed detection of dependent files in the file detection queue more flexible and more practical, the batch executor periodically performs file arrival detection on the dependent files in the file detection queue, and can also be: performing the i-th file arrival detection on the dependent files in the file detection queue, and obtaining the current detection time used for the i-th file arrival detection; wherein i is a positive integer, and i≥1; re-determining a new detection interval based on the detection interval, the current system time and the current detection time; if the new detection interval is less than the preset time threshold, performing the i+1-th file arrival detection on the dependent files in the file detection queue.
[0072] Among them, the current detection time refers to the time required for the batch executor to complete the file arrival detection for all dependent files in the file detection queue for the i-th time. The detection interval time refers to the time interval for performing file arrival detection on the dependent files in the file detection queue; optionally, the detection interval time can be preset according to actual business needs or the experience of technical personnel in this field. For example, the detection interval time can be 60 seconds, and the embodiment of the present invention does not make specific limitations on it. The current system time refers to the system time when the batch executor performs the i-th file arrival detection on the dependent files in the file detection queue. The preset time threshold can be preset according to actual business needs or the experience of technical personnel in this field. For example, the preset time threshold can be 0 seconds, and the embodiment of the present invention does not make specific limitations on it.
[0073] According to the detection interval, the current system time and the current detection time, a new detection interval can be re-determined by calculating the time difference between the current system time and the current detection time as the intermediate time, that is:
[0074] h0=h system -h current ;
[0075] Among them, h0 represents the intermediate time; h system Indicates the current system time; h current Indicates the current detection time. After that, the time difference between the detection interval and the middle time is calculated as the new detection interval, that is:
[0076] h new =h old -h0;
[0077] Among them, h new Indicates the new detection interval; h old Indicates the detection interval.
[0078] Optionally, if the new detection interval is equal to or greater than a preset time threshold, the batch executor performs file arrival detection on dependent files in the file detection queue according to the new detection interval.
[0079] S207: When it is detected that all dependent files have arrived, the target job is processed in a target processing manner.
[0080] The technical solution of the embodiment of the present invention is to parse the target job scheduling instruction after receiving the target job scheduling instruction sent by the distributed batch controller to obtain the instruction parsing content; wherein the instruction parsing content includes the target job identification, target processing method and file dependency of the target job; according to the file dependency, determine at least one dependent file corresponding to the target job; for each dependent file obtained, according to the file identification of the dependent file, obtain the specified arrival position of the dependent file from the file metadata in the distributed batch platform; detect whether the dependent file has reached the specified arrival position through the file detection interface; if so, determine that the dependent file has arrived; if not, determine that the dependent file has not arrived; when it is detected that all dependent files have arrived, use the target processing method to process the target job. In the above technical solution, after the batch executor parses the target job scheduling instruction to obtain the instruction parsing content, it determines at least one dependent file corresponding to the target job according to the file dependency relationship; then, by detecting whether each dependent file has reached its designated arrival position to determine whether each dependent file has arrived, it can more effectively and accurately determine whether each dependent file corresponding to the target job has arrived; then, when it is detected that each dependent file corresponding to the target job has arrived, the target job is processed using the target processing method, avoiding errors caused by the non-arrival of the dependent files corresponding to the target job during the batch executor's processing of the target job, thereby improving the batch executor's processing efficiency for the target job. At the same time, the batch executor uses the file detection interface uniformly configured by the distributed batch platform to detect whether each dependent file corresponding to the target job has arrived, avoiding batch response interruptions caused by the inconsistency of the file detection methods of each batch executor in the distributed batch platform, thereby improving the distributed batch platform's processing efficiency for batch jobs.
[0081] Embodiment 3
[0082] Figure 3 This is a schematic diagram of the structure of a job processing device provided in the third embodiment of the present invention. This embodiment can be applied to the case where a batch executor in a distributed batch platform processes jobs according to received job scheduling instructions. The device can be implemented in the form of hardware and / or software and can be configured in an electronic device with a batch executor. Figure 3 As shown, the device comprises:
[0083] The instruction parsing content determination module 301 is used to parse the target job scheduling instruction after receiving the target job scheduling instruction sent by the distributed batch controller to obtain the instruction parsing content; wherein the instruction parsing content includes the target job identification, target processing mode and file dependency of the target job;
[0084] A dependent file determination module 302, configured to determine at least one dependent file corresponding to a target job according to a file dependency relationship;
[0085] A file arrival detection module 303 is used to perform file arrival detection on each dependent file through a file detection interface; wherein the file detection interface is uniformly configured by the distributed batch platform;
[0086] The job processing module 304 is used to process the target job in a target processing manner when it is detected that all dependent files have arrived.
[0087] The technical solution of the embodiment of the present invention, after receiving the target job scheduling instruction sent by the distributed batch controller, parses the target job scheduling instruction to obtain the instruction parsing content; wherein the instruction parsing content includes the target job identification, target processing method and file dependency of the target job; according to the file dependency, at least one dependent file corresponding to the target job is determined; file arrival detection is performed on each dependent file through the file detection interface; when it is detected that each dependent file has arrived, the target job is processed in the target processing method. In the above technical solution, after the batch executor parses the target job scheduling instruction to obtain the instruction parsing content, it determines at least one dependent file corresponding to the target job according to the file dependency, and when it is detected that each dependent file corresponding to the target job has arrived, it processes the target job in the target processing method, thereby avoiding errors caused by the non-arrival of the dependent files corresponding to the target job during the batch executor's processing of the target job, thereby improving the processing efficiency of the batch executor for the target job. At the same time, the batch executor uses the file detection interface uniformly configured by the distributed batch platform to detect whether the dependent files corresponding to the target job have arrived, avoiding batch response interruptions caused by inconsistent file detection methods of each batch executor in the distributed batch platform, thereby improving the processing efficiency of the distributed batch platform for batch jobs.
[0088] Optionally, the file arrival detection module 303 is specifically used to:
[0089] For each dependent file, according to the file identifier of the dependent file, the specified arrival location of the dependent file is obtained from the file metadata in the distributed batch platform;
[0090] Detect whether the dependent file has reached the specified location through the file detection interface;
[0091] If so, it is determined that the dependent file has been reached;
[0092] If not, it is determined that the dependent file is not reached.
[0093] Optionally, the device further comprises:
[0094] A dependent file storage module is used to store the dependent file into a file detection queue and record the file waiting time after determining that the dependent file has not been reached;
[0095] A prescribed duration acquisition module is used to obtain the prescribed waiting duration and prescribed alarm duration corresponding to the dependent file from the file metadata in the distributed batch platform according to the file identifier of the dependent file;
[0096] The alarm module is used to trigger the alarm mechanism when it detects that the file waiting time exceeds the specified waiting time, send an alarm message to the operation and maintenance personnel, and record the file alarm time;
[0097] The job termination module is used to forcibly terminate the processing of the target job when it is detected that the file alarm duration exceeds the specified alarm duration.
[0098] Optionally, the file metadata in the distributed batch platform is uniformly defined and published by the metadata platform.
[0099] Optionally, the device further comprises:
[0100] The file timing detection module is used to perform file arrival detection on dependent files in the file detection queue at a regular interval.
[0101] Optional, file timing detection module, specifically used for:
[0102] Perform the i-th file arrival detection on the dependent files in the file detection queue, and obtain the current detection time used for the i-th file arrival detection; where i is a positive integer and i≥1;
[0103] Re-determine a new detection interval time according to the detection interval time, the current system time and the current detection time;
[0104] If the new detection interval is less than the preset time threshold, the i+1th file arrival detection is performed on the dependent files in the file detection queue.
[0105] The job processing device provided in the embodiment of the present invention can execute the job processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each job processing method.
[0106] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.
[0107] Embodiment 4
[0108] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0109] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0110] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0111] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a job processing method.
[0112] In some embodiments, the job processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the job processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the job processing method in any other appropriate manner (e.g., by means of firmware).
[0113] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0114] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0115] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0116] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0117] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0118] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0119] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0120] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for processing a job, characterized in that: Executed by a batch executor, the method includes: After receiving the target job scheduling instruction sent by the distributed batch controller, the target job scheduling instruction is parsed to obtain instruction parsing content; wherein the instruction parsing content includes the target job identifier, target processing mode and file dependency of the target job; Determine at least one dependent file corresponding to the target job according to the file dependency relationship; Performing file arrival detection on each dependent file through a file detection interface; wherein the file detection interface is uniformly configured by a distributed batch platform; When it is detected that all dependent files have arrived, the target job is processed using the target processing method.
2. The method according to claim 1, characterized in that The file arrival detection of each dependent file through the file detection interface includes: For each dependent file, according to the file identifier of the dependent file, the designated arrival location of the dependent file is obtained from the file metadata in the distributed batch platform; Detecting, through a file detection interface, whether the dependent file has reached the specified arrival position; If so, it is determined that the dependent file has been reached; If not, it is determined that the dependent file is not reached.
3. The method according to claim 2, characterized in that After determining that the dependent file is not reached, it also includes: The dependent file is stored in the file detection queue and the file waiting time is recorded; According to the file identifier of the dependent file, obtaining the prescribed waiting time and the prescribed alarm time corresponding to the dependent file from the file metadata in the distributed batch platform; When it is detected that the file waiting time exceeds the specified waiting time, an alarm mechanism is triggered, an alarm message is sent to the operation and maintenance personnel, and the file alarm time is recorded; When it is detected that the file alarm duration exceeds the specified alarm duration, the processing of the target job is forcibly terminated.
4. The method according to claim 2, characterized in that: The file metadata in the distributed batch platform is uniformly defined and published by the metadata platform.
5. The method according to claim 3, characterized in that: The method further comprises: The dependent files in the file detection queue are periodically subjected to file arrival detection.
6. The method according to claim 5, characterized in that The regularly performing file arrival detection on dependent files in the file detection queue includes: Performing the i-th file arrival detection on the dependent files in the file detection queue, and obtaining the current detection time used for the i-th file arrival detection; wherein i is a positive integer, and i≥1; Re-determine a new detection interval time according to the detection interval time, the current system time, and the current detection time; If the new detection interval is less than the preset time threshold, the i+1th file arrival detection is performed on the dependent files in the file detection queue.
7. A job processing device, characterized in that: Applied to a batch executor, the device comprises: The instruction parsing content determination module is used to parse the target job scheduling instruction after receiving the target job scheduling instruction sent by the distributed batch controller to obtain the instruction parsing content; wherein the instruction parsing content includes the target job identification, target processing mode and file dependency of the target job; A dependent file determination module, used to determine at least one dependent file corresponding to the target job according to the file dependency relationship; A file arrival detection module, used to perform file arrival detection on each dependent file through a file detection interface; wherein the file detection interface is uniformly configured by a distributed batch platform; The job processing module is used to process the target job using the target processing method when it is detected that all dependent files have arrived.
8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the job processing method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the job processing method according to any one of claims 1 to 6 when executed.
10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the job processing method according to any one of claims 1 to 6.