Job processing method and apparatus for batch processing job, and electronic device

Abstract

The embodiment of the invention provides a job processing method and device for batch processing jobs and electronic equipment, and relates to the field of big data. According to the method, when a start key value of an opening operation is detected, a target batch processing operation is determined according to the opening operation, and the target batch processing operation comprises the opening operation, at least one batch operation and a closing operation. And traversing the database based on the start key value, and determining a to-be-processed job from the at least one batch job. And performing job processing on the to-be-processed job to obtain a corresponding job key value, re-traversing the database based on the job key value to determine and execute a new to-be-processed job, and determining that the processing of the target batch processing job is completed under the condition of determining that the new to-be-processed job is a closing-in job. According to the method, the key value of each job is stored in the database, the jobs needing to be processed are rapidly determined through the database, all the jobs conforming to the key values are processed at the same time, system resources are fully utilized, and the processing speed is increased.

Description

Technical Field

[0001] The present application relates to the field of big data, and particularly to a method, apparatus, and electronic device for processing batch jobs. Background Art

[0002] In the financial system, in order to ensure the accuracy and consistency of data, it is indispensable to set the dependency relationships between jobs. Through reasonable job dependency arrangements, resource utilization can be optimized, concurrent conflicts can be effectively avoided, business logic can be ensured to execute according to the preset steps, and data errors or logical chaos can be prevented.

[0003] Currently, most batch jobs in the financial system adopt a serial processing mode. In this mode, a batch job must wait for its preceding job to be fully executed before it can be started. This serial execution method ensures that the dependency relationships between jobs are strictly adhered to, thus avoiding data conflicts and logical errors that may be caused by concurrent execution.

[0004] However, since batch jobs are executed serially, the processing duration of the entire batch is equal to the cumulative value of the processing durations of each batch job on the critical execution path. As the business volume and data volume increase, the number of jobs and the data volume on the critical execution path will also increase accordingly. This further exacerbates the performance bottleneck in the serial processing mode, resulting in a longer batch processing time and lower efficiency. Summary of the Invention

[0005] The present application provides a method, apparatus, and electronic device for processing batch jobs, aiming to solve the problems brought about by the serial processing method in the existing batch processing, especially the technical problem of long batch processing time and low efficiency in the case of a large number of batch jobs.

[0006] In a first aspect, the present application provides a method for processing batch jobs, including:

[0007] When detecting the start key value of the opening job, determining a target batch job according to the opening job, where the target batch job includes: an opening job, at least one batch job, and a closing job;

[0008] Traversing a database based on the start key value to determine a job to be processed from at least one of the batch jobs, where the database is used to store multiple status key values indicating the start of job processing; the start key value is equal to one of the status key values in the database;

[0009] Performing job processing on the job to be processed to obtain a corresponding job key value, and re-traversing the database based on the job key value to determine and execute a new job to be processed;

[0010] When it is determined that the new job to be processed is the closing operation, it is determined that the target batch job has been processed.

[0011] In a second aspect, the present application provides a job processing device for batch jobs, including:

[0012] A determination module, configured to, when detecting a start key value of an opening operation, determine a target batch job according to the opening operation, where the target batch job includes: an opening operation, at least one batch job, and a closing operation;

[0013] The determination module is further configured to traverse a database based on the start key value, and determine a job to be processed from at least one of the batch jobs, where the database is used to store multiple status key values indicating the start of job processing; the start key value is equal to one of the status key values in the database;

[0014] A processing module, configured to perform job processing on the job to be processed to obtain a corresponding job key value, and re-traverse the database based on the job key value to determine and execute a new job to be processed;

[0015] The determination module is further configured to, when it is determined that the new job to be processed is the closing operation, determine that the target batch job has been processed.

[0016] In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores computer execution instructions;

[0018] The processor executes the computer execution instructions stored in the memory to implement the job processing method for batch jobs as described in the first aspect and various possible implementation manners of the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, on which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the job processing method for batch jobs as described in the first aspect and various possible implementation manners of the first aspect.

[0020] In a fifth aspect, the present application provides a program product, including a computer program, and when the computer program is executed by a processor, it implements the job processing method for batch jobs as described above.

[0021] The job processing method, device, and electronic device for batch jobs provided by this application, after detecting the start key value of the opening job, determine a target batch job sequence based on the opening job, and this sequence includes the opening job, at least one batch job, and the closing job. Use the start key value to retrieve in the database storing the job processing status key values to determine the job to be processed. Subsequently, perform a processing operation on the job to be processed and generate a corresponding job key value. Based on this newly generated job key value, query the database again to determine and execute the next job to be processed. This process is repeated until it is recognized that the job to be processed is the closing job, at which point it indicates that the target batch job sequence has been completed. This method stores the processing status key values of each job through the database, realizes rapid positioning and processing of jobs meeting specific conditions, and thus can more effectively utilize system resources and improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0023] Figure 1 Flow schematic of a job processing method for batch jobs provided by this application Figure 1 ;

[0024] Figure 2 Flow schematic of a job processing method for batch jobs provided by this application Figure 2 ;

[0025] Figure 3 Flow schematic of a job processing method for batch jobs provided by this application Figure 3 ;

[0026] Figure 4 Structural schematic diagram of a job processing device for batch jobs provided by this application;

[0027] Figure 5 Structural schematic diagram of a job processing device for batch jobs provided by this application.

[0028] Through the above accompanying drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. Moreover, the processing of relevant data, such as collection, storage, use, processing, transmission, provision, disclosure, and application, complies with relevant laws, regulations, and standards, takes necessary confidentiality measures, does not violate public order and good customs, and provides corresponding operation entrances for users to choose to authorize or refuse.

[0031] Furthermore, the present application involves big data analysis of user information (including but not limited to personal biometric characteristics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and uses artificial intelligence technology for automated decision-making. For technical solutions that make decisions having a significant impact on personal rights and interests based on the results of automated decision-making, corresponding operation entrances are provided for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, an expert decision-making process will be entered.

[0032] It should be noted that the job processing method, apparatus, and electronic device for batch processing jobs provided in the present application can be used in the field of big data or any field other than big data. The application fields of the job processing method, apparatus, and electronic device for batch processing jobs in the present application are not limited.

[0033] First, the nouns appearing in the present application will be explained:

[0034] Financial system: The financial system in the present application refers to an integrated system that faces the financial industries such as banks and securities, uses computer hardware and software and network devices to collect and process financial information, and continuously provides financial information services.

[0035] Batch processing: It is an offline processing method, also known as batch job processing (Batch Process), which refers to the processing link of non-real-time services in the financial system and consists of opening jobs, closing jobs, and several other business batch jobs between the opening and the closing.

[0036] Job Dependency: It refers to the relationship in a series of batch jobs where the execution of subsequent batch jobs depends on the execution status of previous batch jobs, that is, the association relationship between batch jobs.

[0037] Batch Processing Framework: It refers to a batch processing system that can provide batch scheduling capabilities and schedule batch jobs based on the settings of batch scheduling. In the batch processing process, there are dependency relationships between various jobs, and specific jobs also have requirements for characteristics such as start time and whether they can be run repeatedly. Batch scheduling refers to the mechanism that meets such settings.

[0038] In the financial system, the accuracy and consistency of data are the cornerstones of business continuity and correct decision-making. To ensure the accuracy and consistency of data, the execution of business logic needs to follow a strict order, that is, after one job is completed, its subsequent dependent jobs can start to execute. This dependency relationship between jobs is set based on business requirements and the inherent requirements of the data processing process. Batch jobs, as one of the main ways of data processing in the financial system, automatically execute a series of data processing tasks through a predefined job sequence, so as to meet the high requirements of the business for data accuracy and consistency.

[0039] Currently, most batch jobs in the financial system adopt a serial processing mode. In this mode, a batch job must wait until its previous job is completely executed before it can start. This serial execution method ensures that the dependency relationship between jobs is strictly observed, thus avoiding data conflicts and logical errors that may be caused by concurrent execution.

[0040] However, the serial processing mode of batch jobs results in a long processing time and low resource utilization rate for the entire batch processing process. Moreover, as the business volume and data volume increase, the number of jobs and data volume on the critical execution path increase, further exacerbating the performance bottleneck, significantly reducing the batch processing efficiency, and at the same time limiting the flexibility of job execution.

[0041] In response to the above problems, this application proposes a job processing method for batch jobs. By adjusting the dependency relationship between jobs, the original serial relationship is adjusted to a flat parallel relationship. Each batch job for business processing can be connected to an opening job in the front and a receiving job in the back. And the batch job is further divided into multiple batch jobs, a status transition field is added to each batch job, a key value is generated according to the status transfer field, and after the key value is detected, the job processing is directly performed on the batch job corresponding to the key value, without waiting for the batch jobs at the same level to be processed together, improving the job processing efficiency of batch jobs.

[0042] This application is specifically applied to the efficient data processing process in the financial system, especially the banking system. For example, it can be applied to the end-of-day settlement. During the end-of-day settlement process, a large number of transaction data, account balance updates, interest calculations, and other operations need to be processed. By processing these operations in parallel and monitoring the status transfer fields in real time, the bank can significantly shorten the settlement time and improve customer satisfaction.

[0043] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0044] Figure 1 It is a flow schematic diagram of a job processing method for a batch job provided by an embodiment of this application Figure 1 . As Figure 1 shown, the job processing method for the batch job provided by this embodiment includes:

[0045] S101: When detecting the start key value of the opening job, determine the target batch job according to the opening job.

[0046] Among them, the target batch job includes: an opening job, at least one batch job, and a closing job. The opening job is the initial stage when the batch job is created and submitted to the system for processing, and the closing job is the last stage or the finishing work before the batch job is executed.

[0047] It can be understood that the opening job will start on a pre-set date. After starting, a start key value will be generated to inform the system that the corresponding batch job can start to be processed. When the batch processing system detects the start key value, it marks the start of the batch job. The start key value is a unique set of specific data combinations used to trigger the execution of the uniquely corresponding batch job. Once the start key value is detected, the system will determine the target batch job from a group of possible batch jobs according to the preset rules or logic. This target batch job is a complete job process, including the beginning part (opening job), one or more jobs that need to be processed in batches (batch jobs), and the ending part (closing job). By way of example, the start key value can be the date corresponding to the batch job or other serial numbers. For example, the start key value can be 20241211DAYENDA.

[0048] S102: Traverse the database based on the start key value and determine the jobs to be processed from at least one batch job.

[0049] Among them, the database is used to store multiple status key values indicating the start of job processing; the start key value is equal to one of the status key values in the database.

[0050] It can be understood that after determining the target batch job, the system traverses a database dedicated to storing job status information according to the start key value. This database contains multiple status key values, and each status key value corresponds to the start processing status of a job. By comparing the start key value with the status key values in the database, the system can determine which batch jobs are to be processed. Only when the start key value matches one of the status key values in the database, the corresponding batch job will be regarded as a job to be processed. This process ensures that only jobs in the correct status are selected and executed.

[0051] S103: Process the job to be processed to obtain the corresponding job key value, and re-traverse the database based on the job key value to determine and execute a new job to be processed.

[0052] It can be understood that the system will perform actual job processing on the selected job to be processed. The processing process can include operations such as data reading, processing, conversion, or storage. During the processing, the system will generate or obtain a job key value related to the job, and this key value can include information such as the status, progress, or result of the job. Then, the system will re-traverse the database based on this new job key value to find and determine the next job to be processed. This process may be repeated multiple times until all batch jobs are processed.

[0053] S104: When it is determined that the new job to be processed is a closing job, it is determined that the target batch job is processed.

[0054] It can be understood that during each process of traversing the database and determining the new job to be processed, the system will check whether this job is a closing job. If it is determined that the new job to be processed is a closing job, it means that all jobs in the target batch job have been processed. At this time, the system can consider that the target batch job has been processed. This step marks the end of the entire batch processing process, and the system can also perform some additional operations, such as sending notifications, updating status, or releasing resources.

[0055] A job processing method for batch jobs provided in this embodiment. When the start key value of an opening job is detected, a target batch job is determined according to the opening job. The target batch job includes: an opening job, at least one batch job, and a closing job. Traverse the database based on the start key value to determine the job to be processed from at least one batch job. The database is used to store multiple status key values indicating the start of job processing; the start key value is equal to one of the status key values in the database. Process the job to be processed to obtain the corresponding job key value, and re-traverse the database based on the job key value to determine and execute the new job to be processed. When it is determined that the new job to be processed is a closing job, it is determined that the target batch job processing is completed. This method saves the processing identifier of each job, that is, the key value, in the database, quickly determines the jobs that need to be processed through the database, and processes the jobs that meet the identifier simultaneously, which can make more full use of system resources and improve the processing speed.

[0056] Figure 2 It is a flow schematic diagram of a job processing method for batch jobs provided in an embodiment of this application. Figure 2 As Figure 2 shown, on the basis of the Figure 1 embodiment, a detailed description of the job processing method for batch jobs is given, including:

[0057] S201: Obtain data tables of multiple jobs in the target batch job.

[0058] Among them, the data table is used to indicate the processing timing and process of the job.

[0059] It can be understood that in a batch job environment, each job usually has one or more related data tables. These data tables record the key information of the job, including its processing timing (i.e., when it should start) and processing process (i.e., the specific steps or logic of job execution and the results after processing). When a target batch job needs to be executed, the system first needs to obtain the data tables of the job and all its related jobs. These data tables may be stored in a database or exist in other forms, such as Comma-Separated Values (CSV) files, Excel spreadsheets, or data structures in memory. By reading these data tables, the system can understand the specific requirements of each job, including when they should be triggered, which steps to execute, and how to process input and output data.

[0060] S202: Perform extraction processing on multiple data tables to obtain multiple status key values.

[0061] Among them, the status key value is used to indicate the identifier for starting job processing.

[0062] It is understandable that after obtaining the data tables of multiple jobs in the target batch job, the system needs to extract and process this data. The purpose of this step is to extract key status key values from the data tables, and these status key values are used to identify when the job starts to be processed. The status key values can be a certain column or row in the data table, or can be obtained by calculating or transforming the information in the data table. These status key values are crucial for the scheduling and execution of batch jobs because they tell the system when the processing flow of the job should be started. In practical applications, the status key values represent the priority, dependencies, trigger conditions or any other information related to the start of job processing.

[0063] S203: Save multiple status key values into the database.

[0064] It is understandable that after extracting multiple status key values, the system needs to save them into a database for retrieval and use in subsequent processing. This database can be, for example, a shared non-relational database (Not Only SQL, abbreviated as NoSQL database). The advantage of saving status key values into the database is that they can be quickly retrieved when needed, which is crucial for the scheduling and execution of batch jobs. In addition, saving status key values in the database can also provide persistent storage, so that even if the system fails or restarts, the status key values will not be lost. In practical applications, the design of the database may need to consider factors such as the uniqueness, indexing and query performance of status key values to ensure that the system can efficiently process a large number of batch jobs. At the same time, the design of the database is easier to horizontally scale the system by adding processing nodes, improving the scalability and elasticity of the system. When additional batch jobs are needed, only by adding the status key values corresponding to the batch jobs to the database can the execution of the batch jobs be ensured.

[0065] S204: When detecting the start key value of the opening job, determine the target batch job according to the opening job.

[0066] Among them, step S204 is similar to step S101 and will not be elaborated here.

[0067] S205: Traverse the database based on the start key value and determine the job to be processed from at least one batch job.

[0068] Among them, step S205 is similar to step S102 and will not be elaborated here.

[0069] S206: Determine the processing steps of the job to be processed according to the data table of the job to be processed.

[0070] It is understandable that the data table of the job to be processed indicates the specific steps or logic of job execution and the results after processing. After determining the job to be processed, the data table is analyzed and extracted to determine the processing steps of the job to be processed, facilitating subsequent job processing.

[0071] S207: Perform job processing on the job to be processed according to the processing steps and generate a job key value corresponding to the job to be processed.

[0072] It is understandable that after obtaining the job steps of the job to be processed, the data in the job to be processed is processed and a job key value is spliced according to business elements. Business elements refer to the main business corresponding to the batch job. The business elements need to indicate that the job to be processed and the target batch job are the same, facilitating subsequent extraction and use of data for each batch job by the staff. The job key value is used to indicate that the job to be processed has been completed, that is, the job key value is an identifier for the completion of the job processing process of the job to be processed.

[0073] There is an order relationship between the data of batch jobs in the batch processing job, that is, the job processing process of some batch jobs depends on the data generated after the processing of the previous batch job. Therefore, the role of the job key value is also to remind subsequent batch jobs that the data they need has been generated and the batch jobs that need this data can start the job processing process.

[0074] S208: Determine whether the job key value is equal to the end key value. If not, execute step S209; if so, execute step S210.

[0075] It is understandable that after each job to be processed generates a job key value, it is necessary to first determine whether the job key value is equal to the end key value. The end key value is a specific identifier that has a corresponding relationship with the start key value and is provided by the closing job in the target batch processing job. If the current job key value is not equal to the end key value, it means that there are still batch jobs to be processed, and the database is traversed again to determine a new job to be processed. If the current job key value is equal to the end key value, it means that the end of the batch processing job has been reached.

[0076] S209: Traverse the database again based on the job key value to determine a new job to be processed, and perform job processing on the new job to be processed to obtain a new job key value until the job key value is equal to the end key value.

[0077] It can be understood that when the current job key value is not equal to the end key value, the system will search in the database based on the current job key value to determine the key value stored in the database that is equal to the current job key value, so as to find the new job to be processed, process it, generate a new job key value, and then judge the job key value, repeating this search and generation process until the system finds a job key value equal to the end key value, which indicates that all jobs in the batch job have been processed.

[0078] S210: Determine that the job to be processed is a closing job.

[0079] It can be understood that if the job key value is equal to the end key value, it means that the next job to be processed is a closing job. However, the closing job usually does not require data processing. The system can directly consider that the batch job is about to be completed, and only the closing job needs to seal the batch job to facilitate the subsequent extraction of the target batch job.

[0080] S211: Determine that the target batch job is completed according to the closing job.

[0081] It can be understood that once the system determines that the current job to be processed is a closing job, it will mark the target batch job as completed. This means that all jobs in the batch job have been executed in the preset order, including all batch jobs and closing jobs. The system may update the status information in the database, send notifications to relevant users or system components, or perform other operations related to the completion of the batch job.

[0082] For example, the execution process of this embodiment will be described by way of example below;

[0083] Suppose the status key values of all batch jobs have been saved in the database. The target batch job includes: an opening job, batch job 1, batch job 2, and a closing job. First, obtain the data table of each job. The data table records the status key value at which each job can start processing, the execution data, and the execution steps. The preset execution date of the opening job is December 12, 2024. After starting to execute on the corresponding date, the start key value generated according to the preset conditions is 20241212A. The status key value of batch job 1 is 20241212A, the status key value of batch job 2 is ABC1, and the end key value of the closing job is 20241212B. The above status key values have been saved in the database.

[0084] The opening operation was started on time on December 12, 2024, and 20241212A was generated. The system detected 20241212A, determined the corresponding batch job for the opening operation that met the criteria based on 20241212A, and started searching in the database. Batch job 1 was matched, and the execution started according to the data table of batch job 1, generating job key value ABC1. The job key value ABC1 was not equal to the end key value. Based on ABC1, the database was traversed again, batch job 2 was matched, and the execution started according to the data table of batch job 2, generating job key value 20241212B. The job key value was equal to the end key value, and it was determined that all batch jobs had been fully executed. The closing work of the entire batch process was carried out according to the data table of the closing operation.

[0085] A job processing method for batch jobs provided in this embodiment determines the processing timing and process of jobs by obtaining job data tables. By extracting the status key values in the data tables and storing them in the database, it provides an identifier for subsequent job processing. When the start key value of the opening operation is detected, the target batch job sequence is determined, which consists of the opening operation, several batch jobs, and the closing operation. Subsequently, the first job to be processed is matched and determined in the database using the start key value. According to the data table of the job to be processed, the processing steps are clarified and the job processing is implemented, and the job key value is generated according to the preset rules. If the job key value is equal to the end key value, it is determined as the closing operation, indicating the completion of the batch job; otherwise, the next job is matched and processed based on the job key value until the end key value is reached. This method realizes the automated management and sequential execution of batch jobs through the matching and processing of status key values and job key values, significantly improving the efficiency and accuracy of job processing.

[0086] Figure 3 It is a flow schematic of a job processing method for batch jobs provided in an embodiment of this application Figure 3 As Figure 3 shown, on the basis of the Figure 1 embodiment, a possible implementation method for constructing the data table of each job is described in detail, including:

[0087] S301: Obtain the job processing data of the batch job to be implemented.

[0088] It is understandable that obtaining data is a crucial step before the start of a batch job. The batch jobs to be executed are those that can perform batch processing. These jobs do not necessarily need to be processed simultaneously and can be classified into multiple batch jobs. However, as a batch processing system, it is necessary to know the job processing data of each job for subsequent processing. In the batch processing flow, this data is the basis for subsequent processing and may be stored in a database, file server, cloud storage, or other data sources. This job processing data will indicate the specific execution steps and timing of the batch job, as well as the purpose of job execution and the use of the data generated by job execution.

[0089] S302: Parse and split the job processing data to obtain at least one batch job.

[0090] It is understandable that processing job data means dividing the data into the smallest units. Parsing refers to understanding the structure and meaning of the data so that it can be processed correctly. Splitting means dividing the data into smaller, manageable parts that can be processed independently as batch jobs. By parsing and splitting, a large task can be decomposed into multiple small, parallel jobs, thereby improving processing efficiency and flexibility. Each batch job may contain different data processing tasks, such as data cleaning, transformation, aggregation, or loading into a target system. Ultimately, these batch jobs will jointly complete the entire batch processing flow.

[0091] S303: Determine whether the number of batch jobs is one. If so, execute step S304; if not, execute step S307.

[0092] It is understandable that the number of batch jobs mainly depends on data processing and usage. When the data processing requirements for a certain category are relatively simple and the data volume is not large, only one batch job may be sufficient to complete all processing tasks. Correspondingly, when the data processing requirements for the same category are complex, involving multiple steps or multiple data sets, the task may need to be split into multiple batch jobs for processing. Each batch job is responsible for completing specific processing tasks to ensure the accuracy and efficiency of the overall process.

[0093] S304: Construct an opening job and a closing job.

[0094] It is understandable that if there is only one batch job, it means that the data scheduling and processing of this job are relatively simple, or rather, the processing of this job does not need to depend on other jobs and can be completed independently. Therefore, a batch job is constructed based on this batch job to complete the processing of this batch job in advance. If the data generated by this batch job needs to be used subsequently, it can be directly called without having to process this batch job when the data is needed, thus improving the efficiency of batch processing.

[0095] When generating the batch job corresponding to the batch operation, it is necessary to construct an opening job and a closing job for it to seal the batch operation and ensure the non-disclosure of the data of the batch operation. The role of the opening job is to display the data usage of the batch operation and indicate the timing when the batch operation starts to be processed. The purpose of the closing job is to indicate that the internal processing of the batch operation is completed and the data can be called.

[0096] S305: Establish the association relationships between the opening job and the batch job, and between the closing job and the batch job respectively.

[0097] It can be understood that when constructing the batch job process, in order to ensure the correct connection and sequential execution between each job, it is necessary to establish the association relationships between the opening job and the batch job, and between the closing job and the batch job respectively. The establishment of this association relationship is essentially a clear definition of the job processing timing. It ensures that the opening job starts first, indicating the direction for the data processing and usage of the batch job; subsequently, the batch job is processed under the guidance of the opening job; finally, the closing job is executed after the batch job is completed, marking the successful end of the internal processing flow of the entire batch job, and at this time the data can be safely called. This process strictly follows the processing sequence of the opening job, the batch job, and the closing job, and each job can enter the next link only after completion, thus ensuring the efficiency and accuracy of the batch job.

[0098] S306: Generate the data tables of the opening job, the closing job, and the batch job respectively according to the association relationship and the job processing data.

[0099] It can be understood that the association relationship refers to the logical connection between the opening job, the batch job, and the closing job in terms of execution sequence, data transfer, and dependency relationship. The job processing data are the data such as the processing steps and processing requirements of each job. Generating each data table for subsequent processing jobs can quickly clarify the processing requirements of the jobs and perform accurate and complete processing of the jobs. The opening job data table is mainly used to store the relevant information of the opening job, including but not limited to job ID, job name, start time, data usage description, start key value, etc. The batch job data table mainly records the execution situation of the batch job, including job ID, input data details, processing logic description, output results, etc. The closing job data table is mainly used to record the execution results of the closing job, including job ID, job name, completion time, data output location, status flag, end key value, etc.

[0100] S307: Sort multiple batch jobs according to the job processing data to obtain the processing sequence of the batch jobs.

[0101] It is understandable that when there are multiple batch jobs to be executed, we need to sort them reasonably according to factors such as the data type, priority, data volume, and processing time of each job. This sorting process will determine the execution order of each batch job to ensure the efficiency and accuracy of the batch processing flow. The sorted order will be the basis for subsequent job scheduling and execution.

[0102] S308: Determine the first batch job and the second batch job according to the processing order.

[0103] It is understandable that after determining the processing order of the batch jobs, we can classify the jobs into two categories according to this order: the first batch job and the second batch job. The first batch job is the job that will be executed first after sorting, which marks the start of the entire batch processing flow. The second batch jobs are the jobs that come after the first batch job in the sorting and will be executed sequentially after the first batch job is completed.

[0104] S309: Construct an opening job and generate the start key value of the opening job and the data table of the opening job according to the preset generation conditions.

[0105] It is understandable that the opening job is an important link in the batch processing flow. It is responsible for starting the entire batch processing flow and setting the execution conditions for subsequent jobs. When constructing the opening job, we need to generate the start key value according to the preset generation conditions (such as set time, event trigger, etc.). This key value will be used to indicate the start of the batch processing flow and the batch jobs can start to execute. At the same time, we also need to create a data table for the opening job to record relevant information during its execution, such as start time, execution status, etc.

[0106] S310: Establish an association relationship between the opening job and the first batch job based on the start key value.

[0107] It is understandable that in order to ensure that the first batch job can be executed immediately after the opening job is started, we need to establish an association relationship between them. This association relationship is realized based on the start key value, that is, when the opening job is executed and generates the start key value, this key value will be used to trigger the execution of the first batch job. In this way, we can ensure the sequential execution and correct connection between the opening job and the first batch job.

[0108] S311: Establish the association relationships between the first batch job and the second batch job, and between multiple second batch jobs respectively according to the preset status transition fields.

[0109] It is understandable that in the batch processing process, the execution status of each job will affect the execution of subsequent jobs. Therefore, we need to establish the association relationships between the first batch of jobs, the second batch of jobs, and multiple second batch of jobs according to the preset status transition fields. Such association relationships will ensure that each job is triggered and executed at the correct time point, while ensuring that the data transfer and dependency relationships between jobs are correctly processed.

[0110] S312: Generate data tables for the first batch of jobs and the second batch of jobs respectively according to the association relationships and job processing data.

[0111] It is understandable that we need to generate data tables for the first batch of jobs and the second batch of jobs respectively according to the established association relationships and the data content processed by each job. These data tables will be used to record the execution timing, processing steps, and related metadata of each job. Through these data tables, we can clarify when each job should be processed and what the results are after processing.

[0112] Optionally, the process of constructing the closing job is described in detail as follows:

[0113] Perform screening processing on the second batch of jobs to obtain the third batch of jobs, and the third batch of jobs is the last batch of jobs to start processing;

[0114] It is understandable that the second batch of jobs will be executed in sequence and generate corresponding results. After the third batch of jobs, that is, the last batch of jobs, is executed, that is, when the batch processing process is completed, an identification information is required to inform the system that the batch processing process is completed.

[0115] Construct a closing job and generate an end key value of the closing job according to the preset generation conditions.

[0116] It is understandable that the end key value is used to indicate the completion of the batch processing job. The preset generation conditions for generating the end key value of the closing job are the same as those for generating the start key value of the opening job, both of which are generated by the set time and event triggers. The closing job is the last link in the batch processing process, and it is responsible for processing all the remaining data or tasks generated during the batch processing process. By constructing the closing job, the integrity of the entire batch processing process can be ensured, and the situation of data omission or uncompleted tasks can be avoided. There is a corresponding relationship between the end key value and the start key value, which can guarantee the integrity of the batch processing job and prevent job omission.

[0117] Establish the association relationship between the closing job and the third batch of jobs based on the end key value.

[0118] It can be understood that in order to ensure that after the execution of the third batch of jobs, the closing job is immediately executed, which is the final sealing work of the complete batch job, it is necessary to establish an association relationship between the third batch of jobs and the closing job.

[0119] Add the association relationship to the data table of the third batch of jobs, so that after the processing of the third batch of jobs is completed, the closing job can be continued to be executed, and the processing process of the batch job can be completed.

[0120] It can be understood that adding the association relationship to the data table of the third batch of jobs is also adding the end key value to the data table of the third batch of jobs, so that after the third batch of jobs is executed, an end key value can be generated to remind the closing job to execute and complete the processing process of the batch job.

[0121] A job processing method for a batch job provided in this embodiment. The method obtains job processing data for batch jobs to be implemented and parses the job processing data into at least one batch job. When there is a single batch job, an opening job and a closing job are added, and their associations with the batch job are established to generate their respective data tables. When there are multiple batch jobs, according to the sorting of the job processing data, the first and subsequent batch jobs are determined. Then, an opening job is constructed, and a start key value and a data table are generated and associated with the first job. Then, using a preset status transition field, associations between the first and subsequent, and subsequent batch jobs are established, and corresponding data tables are generated. This method realizes the efficient and orderly management of batch jobs by systematically constructing the association relationships of opening, batch, and closing jobs and generating detailed data tables, improves the automation degree and accuracy of job execution, and enhances the overall job processing efficiency.

[0122] Figure 4 It is a structural schematic diagram of a job processing device for a batch job provided in this application. As Figure 4 shown, this application provides a job processing device for a batch job. The job processing device 400 for the batch job includes:

[0123] A determination module 401, configured to determine a target batch job according to the opening job when detecting a start key value of the opening job. The target batch job includes: an opening job, at least one batch job, and a closing job;

[0124] The determination module 401 is further configured to traverse the database based on the start key value and determine a job to be processed from at least one of the batch jobs. The database is used to store multiple status key values indicating the start of job processing; the start key value is equal to one of the status key values in the database;

[0125] The processing module 402 is configured to process the to-be-processed job to obtain a corresponding job key value, and re-traverse the database based on the job key value to determine and execute a new to-be-processed job;

[0126] The determining module 401 is further configured to determine that the target batch processing job is completed when determining that the new to-be-processed job is the closing job.

[0127] Optionally, the device further includes: an obtaining module 403 and a saving module 404;

[0128] The obtaining module 403 is configured to obtain data tables of multiple jobs in the target batch processing job, and the data tables are used to indicate the processing timing and process of the jobs;

[0129] The processing module 402 is further configured to perform extraction processing on the multiple data tables to obtain multiple status key values, and the status key values are used to indicate the identifier for starting to process the job;

[0130] The saving module 404 is configured to save the multiple status key values into the database.

[0131] Optionally, the device further includes: a building module 405 and a generating module 406;

[0132] The obtaining module 403 is further configured to obtain job processing data of the to-be-executed batch processing job;

[0133] The processing module 402 is further configured to perform parsing and splitting processing on the job processing data to obtain at least one batch job;

[0134] The building module 405 is configured to build an opening job and a closing job when there is one batch job;

[0135] The building module 405 is further configured to respectively establish an association relationship between the opening job and the batch job, and between the closing job and the batch job;

[0136] The generating module 406 is configured to respectively generate data tables of the opening job, the closing job, and the batch job according to the association relationship and the job processing data.

[0137] Optionally, the processing module 402 is further configured to, when there are multiple batch jobs, perform sorting processing on the multiple batch jobs according to the job processing data to obtain the processing order of the batch jobs;

[0138] The determining module 401 is further configured to determine a first batch job and a second batch job according to the processing sequence; the first batch job is the batch job that starts to be processed first, and the second batch job is the remaining batch jobs;

[0139] The constructing module 405 is further configured to construct an opening job, and generate a start key value of the opening job and a data table of the opening job according to a preset generation condition;

[0140] The constructing module 405 is further configured to establish an association relationship between the opening job and the first batch job based on the start key value;

[0141] The constructing module 405 is further configured to establish an association relationship between the first batch job and the second batch job, and between multiple second batch jobs respectively according to a preset status transition field;

[0142] The generating module 406 is further configured to generate data tables of the first batch job and the second batch job respectively according to the association relationship and the job processing data.

[0143] Optionally, the processing module 402 is further configured to perform a screening process on the second batch job to obtain a third batch job, and the third batch job is the batch job that starts to be processed last;

[0144] The constructing module 405 is further configured to construct a closing job, and generate an end key value of the closing job according to a preset generation condition, and the end key value is used to indicate that the batch processing job is completed;

[0145] The constructing module 405 is further configured to establish an association relationship between the closing job and the third batch job based on the end key value;

[0146] The saving module 404 is further configured to add the association relationship to the data table of the third batch job, so as to continue to execute the closing job after the third batch job is processed, and complete the processing process of the batch processing job.

[0147] Optionally, the determining module 401 is further configured to determine the processing steps of the job to be processed according to the data table of the job to be processed;

[0148] The processing module 402 is further configured to perform job processing on the job to be processed according to the processing steps, and generate a job key value corresponding to the job to be processed.

[0149] Optionally, the device further includes: a judging module 407;

[0150] The determination module 407 is configured to determine whether the job key value is equal to the end key value;

[0151] The determination module 401 is further configured to, when the job key value is equal to the end key value, determine that the job to be processed is a closing job;

[0152] The determination module 401 is further configured to, when the job key value is not equal to the end key value, re-traverse the database based on the job key value, determine a new job to be processed, and perform job processing on the new job to be processed to obtain a new job key value until the job key value is equal to the end key value.

[0153] The job processing device for batch processing jobs provided by the embodiments of the present application has an implementation principle and technical effects similar to those of the corresponding parts in the foregoing job processing method for batch processing jobs, and will not be elaborated herein.

[0154] Figure 5 It is a schematic structural diagram of a job processing device for a batch processing job provided by the present application. As Figure 5 shown, the present application provides a job processing device for a batch processing job. The job processing device 500 for the batch processing job includes: a receiver 501, a transmitter 502, a processor 503, and a memory 504.

[0155] The receiver 501 is configured to receive instructions and data;

[0156] The transmitter 502 is configured to transmit instructions and data;

[0157] The memory 504 is configured to store computer execution instructions;

[0158] The processor 503 is configured to execute the computer execution instructions stored in the memory 504 to implement each step performed by the job processing method for batch processing jobs in the foregoing embodiments. For details, reference may be made to the relevant descriptions in the embodiments of the job processing method for batch processing jobs.

[0159] Optionally, the foregoing memory 504 may be either independent or integrated with the processor 503.

[0160] When the memory 504 is independently provided, the electronic device further includes a bus for connecting the memory 504 and the processor 503.

[0161] The implementation principle and technical effects of the electronic device provided in this embodiment may be referred to the foregoing embodiments, and will not be elaborated herein.

[0162] The embodiments of the present application also provide a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in any of the foregoing embodiments is implemented.

[0163] The embodiments of the present application also provide a computer program product, including a computer program, which implements the method described in any of the foregoing embodiments when executed by a processor.

[0164] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0165] Furthermore, it should be noted that although the steps in the flowchart are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0166] It should be understood that the above device embodiments are illustrative, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0167] In addition, unless otherwise specified, in each embodiment of the present application, each functional unit / module can be integrated in one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0168] In the above embodiments, the descriptions of the various embodiments each have their own emphasis. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0169] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0170] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A method for processing a batch job, characterized in that: include: When the start key value of the opening operation is detected, a target batch processing operation is determined according to the opening operation, wherein the target batch processing operation includes: an opening operation, at least one batch operation and a closing operation; Traversing a database based on the start key value to determine a job to be processed from at least one of the batch jobs, the database being used to store a plurality of status key values ​​indicating the start of job processing; the start key value being equal to a status key value in the database; Performing job processing on the pending job to obtain a corresponding job key value, and re-traversing the database based on the job key value to determine and execute a new pending job; When it is determined that the new job to be processed is the closing job, it is determined that the processing of the target batch job is completed.

2. The method according to claim 1, characterized in that Before traversing the database based on the startup key value to determine a job to be processed from at least one of the batch jobs, the method further includes: Acquire a data table of multiple jobs in a target batch job, wherein the data table is used to indicate a processing time and a processing process of the job; Extracting and processing the plurality of data tables to obtain a plurality of status key values, wherein the status key values ​​are used to indicate an identification of starting processing of the job; The plurality of state key values ​​are saved in a database.

3. The method according to claim 2, characterized in that Before obtaining the data tables of multiple jobs in the target batch job, the method further includes: Obtaining job processing data of a batch job to be executed; Parsing and splitting the job processing data to obtain at least one batch job; When there is one batch operation, construct an opening operation and a closing operation; Establishing associations between the opening operation and the batch operation, and between the closing operation and the batch operation respectively; According to the association relationship and the operation processing data, data tables for the opening operation, the closing operation and the batch operation are generated respectively.

4. The method according to claim 3, characterized in that The method further comprises: In the case where there are multiple batch jobs, the multiple batch jobs are sorted according to the job processing data to obtain a processing order of the batch jobs; According to the processing sequence, a first batch job and a second batch job are determined; the first batch job is the first batch job to be processed, and the second batch job is the remaining batch job; Constructing an opening operation, and generating a start key value of the opening operation and a data table of the opening operation according to preset generation conditions; Establishing an association relationship between the opening operation and the first batch operation based on the start key value; Establishing associations between the first batch job and the second batch job, and between a plurality of the second batch jobs, respectively, according to a preset state transfer field; According to the association relationship and the job processing data, data tables of the first batch job and the second batch job are generated respectively.

5. The method according to claim 4, characterized in that The method further comprises: Screening the second batch of jobs to obtain a third batch of jobs, where the third batch of jobs is the last batch of jobs to be processed; Constructing a closing operation, and generating an end key value of the closing operation according to a preset generation condition, wherein the end key value is used to indicate that the batch operation is completed; Establishing an association relationship between the closing operation and the third batch operation based on the end key value; The association relationship is added to the data table of the third batch job, so that the closing operation is continued after the third batch job is processed, thereby completing the processing process of the batch job.

6. The method according to claim 1, characterized in that The performing job processing on the to-be-processed job to obtain a corresponding job key value includes: Determining the processing steps of the job to be processed according to the data table of the job to be processed; The job to be processed is processed according to the processing steps, and a job key value corresponding to the job to be processed is generated.

7. The method according to claim 5, characterized in that The re-traversing the database based on the job key value to determine and execute a new job to be processed includes: Determine whether the job key value is equal to the end key value; When the job key value is equal to the end key value, determining that the job to be processed is a closing job; When the job key value is not equal to the end key value, the database is re-traversed based on the job key value to determine a new job to be processed, and the new job to be processed is processed to obtain a new job key value until the job key value is equal to the end key value.

8. A job processing device for batch processing jobs, comprising: A determination module, configured to determine a target batch processing operation according to the opening operation when a start key value of the opening operation is detected, wherein the target batch processing operation includes: an opening operation, at least one batch operation and a closing operation; The determination module is further used to traverse a database based on the start key value to determine a job to be processed from at least one of the batch jobs, the database being used to store a plurality of status key values ​​indicating the start of job processing; the start key value is equal to a status key value in the database; A processing module, used to perform job processing on the pending job, obtain a corresponding job key value, and re-traverse the database based on the job key value to determine and execute a new pending job; The determination module is further configured to determine that the target batch processing job is completed when it is determined that the new job to be processed is the closing job.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images