Batch job processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a batch job processing method and device, a storage medium and electronic equipment. The method comprises the steps of obtaining job information of multiple to-be-executed batch jobs, and obtaining database information of a target database of the multiple to-be-executed batch jobs; processing the job information of the plurality of batch jobs and the database information through a target neural network model to obtain priorities corresponding to the plurality of batch jobs and a reference threshold value of a concurrency number of a target database; and adjusting parameters of the target database according to the reference threshold, and controlling the adjusted target database to execute the plurality of batch jobs based on the priorities corresponding to the plurality of batch jobs. Through the method and the device, the problem of low batch job processing efficiency caused by setting a fixed priority for the batch jobs and setting a fixed concurrent number threshold for the database when the batch jobs are processed in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a batch job processing method, device, storage medium and electronic device. Background Art

[0002] With the rapid changes in the financial environment and increasingly stringent regulatory requirements, financial institutions are faced with the challenge of processing massive amounts of data, especially when data reporting shifts from summary data to detailed data, and the reporting frequency increases from monthly to daily. Data processing efficiency becomes critical. At present, in the relevant technology, when executing batch jobs related to the aforementioned data, the strategy of fixing the priority of batch jobs and the concurrent threshold of the database is often adopted. This static job scheduling mode has the problem of low job processing efficiency when facing large amounts of data and high-frequency reporting requirements.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The main purpose of the present application is to provide a batch job processing method, device, storage medium and electronic device to solve the problem in the related art that when processing batch jobs, a fixed priority is set for the batch job and a fixed concurrency threshold is set for the database, resulting in low efficiency in batch job processing.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for processing batch jobs is provided. The method includes: obtaining job information of multiple batch jobs to be executed, and obtaining database information of a target database for executing multiple batch jobs, wherein multiple batch jobs are associated with the same target database, and the database information includes at least the current threshold value and the current concurrency of the target database; processing the job information and database information of multiple batch jobs through a target neural network model to obtain the priorities corresponding to the multiple batch jobs and the reference threshold value of the concurrency of the target database; adjusting the parameters of the target database according to the reference threshold value, and controlling the adjusted target database to execute multiple batch jobs based on the priorities corresponding to the multiple batch jobs.

[0006] Furthermore, the batch job processing method also includes: determining the current resource usage rate of the target database from the database information; obtaining the predicted resource occupancy rate of each batch job on the target database; judging whether the target database meets the preset conditions based on the current resource usage rate and the predicted resource occupancy rate of the target database; and adjusting the parameters of the target database based on the reference threshold when it is determined that the target database meets the preset conditions.

[0007] Furthermore, the batch job processing method also includes: calculating the target resource utilization rate of the target database when executing multiple batch jobs based on the current resource utilization rate and the predicted resource occupancy rate of the target database; when the target resource utilization rate is greater than the preset utilization rate, determining that the target database does not meet the preset conditions; when the target resource utilization rate is less than or equal to the preset utilization rate, determining that the target database meets the preset conditions.

[0008] Furthermore, the batch job processing method further includes: after determining whether the target database meets the preset conditions, if it is determined that the target database does not meet the preset conditions, generating warning information, wherein the warning information indicates that there is a risk of exceeding the limit of resources of the target database.

[0009] Furthermore, the batch job processing method also includes: processing the job information and database information of multiple batch jobs through the target neural network model to obtain the priorities corresponding to the multiple batch jobs, the reference threshold of the concurrency number of the target database, and the predicted resource occupancy rate of each batch job for the target database.

[0010] Furthermore, the batch job processing method also includes: before obtaining the job information of multiple batch jobs to be executed, receiving a data processing program sent by the target object, wherein the data processing program is used to process a data table in at least one database, and the at least one database includes a target database; identifying the data table targeted by the data processing program from the data processing program; and generating batch jobs based on the data table targeted by the data processing program and the data processing program.

[0011] Furthermore, the batch job processing method also includes: generating an initial batch job based on the data table and the data processing program targeted by the data processing program; sending the initial batch job to the target object; and adjusting the initial batch job to obtain the batch job when receiving a job adjustment instruction from the target object.

[0012] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a batch job processing device is provided. The device includes: an acquisition module, which is used to acquire the job information of multiple batch jobs to be executed, and acquire the database information of the target database for executing multiple batch jobs, wherein the multiple batch jobs are associated with the same target database, and the database information at least includes the current threshold value and the current concurrency number of the target database; a first processing module, which is used to process the job information and database information of multiple batch jobs through a target neural network model, and obtain the priorities corresponding to the multiple batch jobs and the reference threshold value of the concurrency number of the target database; a second processing module, which is used to adjust the parameters of the target database according to the reference threshold value, and control the adjusted target database to execute multiple batch jobs based on the priorities corresponding to the multiple batch jobs.

[0013] Furthermore, the second processing module also includes: a determination submodule, used to determine the current resource utilization rate of the target database from the database information; an acquisition submodule, used to obtain the predicted resource utilization rate of each batch job on the target database; a judgment submodule, used to judge whether the target database meets the preset conditions based on the current resource utilization rate and the predicted resource utilization rate of the target database; and a first processing submodule, used to adjust the parameters of the target database according to the reference threshold when it is determined that the target database meets the preset conditions.

[0014] Furthermore, the judgment submodule also includes: a calculation unit, which is used to calculate the target resource utilization rate of the target database when executing multiple batch jobs based on the current resource utilization rate and the predicted resource occupancy rate of the target database; a first determination unit, which is used to determine that the target database does not meet the preset conditions when the target resource utilization rate is greater than the preset utilization rate; and a second determination unit, which is used to determine that the target database meets the preset conditions when the target resource utilization rate is less than or equal to the preset utilization rate.

[0015] Furthermore, the batch job processing device also includes: a first generating module, which is used to generate warning information when it is determined that the target database does not meet the preset conditions, wherein the warning information indicates that there is a risk of exceeding the limit of resources in the target database.

[0016] Furthermore, the first processing module also includes: a second processing sub-module, which is used to process the job information and database information of multiple batch jobs through the target neural network model to obtain the priorities corresponding to the multiple batch jobs, the reference threshold of the concurrency number of the target database, and the predicted resource occupancy rate of each batch job for the target database.

[0017] Furthermore, the batch job processing device also includes: a receiving module, used to receive a data processing program sent by a target object, wherein the data processing program is used to process a data table in at least one database, and at least one database includes a target database; an identification module, used to identify the data table targeted by the data processing program from the data processing program; and a second generation module, used to generate batch jobs based on the data table targeted by the data processing program and the data processing program.

[0018] Furthermore, the second generation module also includes: a generation submodule, which is used to generate an initial batch job based on the data table and data processing program targeted by the data processing program; a sending submodule, which is used to send the initial batch job to the target object; and a third processing submodule, which is used to adjust the initial batch job to obtain a batch job when receiving a job adjustment instruction from the target object.

[0019] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned batch job processing method.

[0020] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, the electronic device comprising a memory storing an executable program; and a processor for running the program, wherein the above-mentioned batch job processing method is executed when the program is running.

[0021] In order to achieve the above objective, according to another aspect of the present application, a computer program product is provided, comprising computer instructions, which implement the steps of the above batch job processing method when executed by a processor.

[0022] In an embodiment of the present application, by obtaining the job information of multiple batch jobs to be executed and obtaining the database information of the target database of the multiple batch jobs to be executed, the job characteristics of the batch jobs to be executed and the real-time working conditions of the database used to execute the jobs are effectively obtained. By using the target neural network model to process the job information and the database information, the priorities corresponding to the multiple batch jobs and the reference threshold of the concurrency of the target database are obtained, and the job priority and the concurrency threshold of the database are dynamically determined according to the job characteristics and the real-time working conditions of the database. Therefore, when executing batch jobs according to the dynamically determined job priority and the concurrency threshold of the database, the operation between jobs can be better coordinated, resource conflicts can be reduced, the waiting time of subsequent jobs that depend on data processing results can be reduced, and the rationality of the use of target database resources can be improved to avoid resource surplus or performance bottlenecks, thereby effectively improving the processing efficiency of batch jobs.

[0023] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining the job priority and the concurrent number threshold of the database when processing batch jobs, realizes the technical effect of improving the processing efficiency of batch jobs, and solves the problem of setting a fixed priority for batch jobs and a fixed concurrent number threshold for the database when processing batch jobs in the related technology, resulting in low efficiency in batch job processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 is a hardware structure block diagram of a computer terminal provided according to an embodiment of the present application;

[0026] Figure 2 is a flowchart of a batch job processing method provided according to an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of adjusting target database parameters according to an embodiment of the present application;

[0028] Figure 4 is a working schematic diagram of a target neural network model provided according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of generating a batch job according to an embodiment of the present application;

[0030] Figure 6 is a working schematic diagram of a target scheduling system provided according to an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of a batch job processing device provided according to an embodiment of the present application;

[0032] Figure 8 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0036] Example 1

[0037] According to an embodiment of the present application, an embodiment of a method for processing a batch job is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a batch job processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the batch job processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the batch job processing method described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] Under the above operating environment, this application provides Figure 2 The processing method of batch jobs is shown. Figure 2 It is a flowchart of a batch job processing method provided according to an embodiment of the present application.

[0044] Step S201, obtaining job information of multiple batch jobs to be executed, and obtaining database information of a target database for the multiple batch jobs to be executed, wherein the multiple batch jobs are associated with the same target database, and the database information at least includes a current threshold value and a current concurrency number of the target database.

[0045] Optionally, electronic devices, application systems, servers and other devices may be used as the execution subject of the present application. In this embodiment, the target scheduling system is used as the execution subject to execute the above-mentioned batch job processing method.

[0046] Optionally, the multiple batch jobs to be executed refer to batch jobs that meet time trigger conditions, or batch jobs that meet event trigger conditions. For batch jobs under time trigger conditions, the execution time of these jobs is pre-set, and is usually closely related to the periodic needs of data processing, such as fixed time points of each day, week, or month. When the system time reaches the scheduled execution time of these jobs, they are considered to be pending. For batch jobs under event trigger conditions, the execution of such jobs is driven by specific events occurring inside or outside the system. For example, when new data arrives, the system status changes, or external events occur, the batch jobs related to these events will be automatically identified as pending to respond to real-time data processing needs or specific business events.

[0047] Optionally, the job information of the batch job includes but is not limited to the job name, job type, database where the job is running, current priority of the job, etc.; the database information includes at least the current threshold of the concurrency number of the target database and the current concurrency number, and the database information may also include the current resource usage rate of the target database, and the current resource usage rate includes but is not limited to the I / O (Input / Output) usage rate and the CPU (Central Processing Unit) usage rate. The aforementioned concurrency number of the target database refers to the number of batch jobs executed simultaneously by the target database.

[0048] Optionally, the aforementioned multiple batch jobs are associated with the same target database, that is, the aforementioned multiple batch jobs are executed through the same target database.

[0049] It should be noted that the content of batch jobs is related to specific application scenarios, and the content of batch jobs in different application scenarios is also different. For example, in the case of financial regulatory reporting, the content of batch jobs is to extract, convert and load detailed data from various business systems to the regulatory reporting platform, so that regulatory agencies can monitor the business activities of financial institutions in real time and assess potential financial risks. These jobs may include specific tasks such as verifying data accuracy, format conversion, data aggregation, and generating reports that meet regulatory requirements. Therefore, in this embodiment, the content of batch jobs is not specifically limited.

[0050] Step S202, processing the job information and database information of the plurality of batch jobs through the target neural network model to obtain the priorities corresponding to the plurality of batch jobs and the reference threshold value of the concurrency number of the target database.

[0051] Optionally, the target neural network model is a pre-trained neural network model. In some embodiments, the input data of the target neural network model includes at least: (1) job information, such as job name, job type, database where the job runs, current priority of the job, etc.; (2) database information, including at least the current threshold of the concurrency of the target database and the current concurrency. The input data of the target neural network model may also include: the current resource usage rate of the target database, and (3) system warning parameters, i.e., the warning threshold of the concurrency of the target database.

[0052] In some embodiments, the output data of the target neural network model includes at least: (1) the job name; (2) the job priority for which adjustment is recommended; (3) the reference threshold value for the concurrency of the target database for which adjustment is recommended. The output data of the target neural network model may also include: (4) the IO and CPU usage resources that may be occupied by the batch job, that is, the predicted resource occupancy rate.

[0053] Optionally, the target neural network model can be trained in the following manner: construct a training sample set, wherein the training samples in the training sample set include job information of a sample batch job, database information of a sample database associated with the sample batch job, the true labels of the training samples include at least the priority corresponding to the sample batch job and a reference threshold of the concurrency of the sample database, and the true labels of the training samples may also include the resource occupancy rate of the sample database by the sample batch job; and train the initial neural network model based on the constructed training sample set to obtain the target neural network model.

[0054] Step S203 , adjusting the parameters of the target database according to the reference threshold, and controlling the adjusted target database to execute the multiple batch jobs based on the priorities corresponding to the multiple batch jobs.

[0055] In traditional scheduling systems, the priority of batch jobs is usually manually pre-set and remains unchanged during the scheduling cycle. However, when faced with large amounts of data and high-frequency reporting requirements, fixed priority settings cannot dynamically respond to changes in the importance and urgency of tasks, resulting in critical and time-sensitive jobs being delayed due to low priority, thereby affecting the timeliness and accuracy of data reporting. At the same time, for the concurrency threshold of the database platform, related technologies usually adopt the method of setting a fixed upper limit to prevent excessive consumption of database resources. However, this strategy ignores the dynamic changes of database resources and the actual needs of jobs. When database resources are abundant, the fixed threshold limits the parallel processing capacity, making it impossible for the system to fully utilize idle resources, reducing the overall processing efficiency. When resources are tight, the fixed concurrency may cause the system to be overloaded, causing performance bottlenecks, affecting data processing stability and response speed. The combination of these two static configuration strategies makes it difficult for related technologies to achieve effective resource utilization and reasonable job scheduling when processing batch jobs, thereby affecting job processing efficiency.

[0056] Therefore, in the present application, after obtaining the priorities corresponding to multiple batch jobs and the reference threshold of the concurrency of the target database through the target neural network model, the parameters of the target database can be adjusted according to the reference threshold to obtain the adjusted target database. For example, based on the reference threshold, the concurrency parameters of the target database are adjusted to limit the concurrency of the target database during the execution of batch jobs.

[0057] Optionally, the target scheduling system can control the adjusted target database to execute multiple batch jobs based on the priorities corresponding to the multiple batch jobs. In some embodiments, before the batch job starts running, the priority of the batch job is adjusted, and the new priority information is stored in a central data storage area accessible to the target database. This storage area can be a job scheduling table or a priority queue. The aforementioned storage area can be a part of the target database, or an external system that is closely integrated with the target database. When the batch jobs are submitted to the target database for execution, their priority information will also be submitted to the storage area for the target database to obtain, so as to execute multiple batch jobs based on the priorities corresponding to the multiple batch jobs. This achieves the elastic contraction of the database's concurrent resources and the automatic optimization of job priorities, which can effectively improve the efficiency of the database in processing batch jobs.

[0058] In an embodiment of the present application, by obtaining the job information of multiple batch jobs to be executed and obtaining the database information of the target database of the multiple batch jobs to be executed, the job characteristics of the batch jobs to be executed and the real-time working conditions of the database used to execute the jobs are effectively obtained. By using the target neural network model to process the job information and the database information, the priorities corresponding to the multiple batch jobs and the reference threshold of the concurrency of the target database are obtained, and the job priority and the concurrency threshold of the database are dynamically determined according to the job characteristics and the real-time working conditions of the database. Therefore, when executing batch jobs according to the dynamically determined job priority and the concurrency threshold of the database, the operation between jobs can be better coordinated, resource conflicts can be reduced, the waiting time of subsequent jobs that depend on data processing results can be reduced, and the rationality of the use of target database resources can be improved to avoid resource surplus or performance bottlenecks, thereby effectively improving the processing efficiency of batch jobs.

[0059] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining the job priority and the concurrent number threshold of the database when processing batch jobs, realizes the technical effect of improving the processing efficiency of batch jobs, and solves the problem of setting a fixed priority for batch jobs and a fixed concurrent number threshold for the database when processing batch jobs in the related technology, resulting in low efficiency in batch job processing.

[0060] Optionally, in the batch job processing method provided in the embodiment of the present application, the parameters of the target database are adjusted according to the reference threshold, including: determining the current resource utilization rate of the target database from the database information; obtaining the predicted resource utilization rate of the target database by each batch job; judging whether the target database meets the preset conditions based on the current resource utilization rate and the predicted resource utilization rate of the target database; and adjusting the parameters of the target database according to the reference threshold when it is determined that the target database meets the preset conditions.

[0061] Optionally, the current resource usage of the target database includes but is not limited to I / O usage and CPU usage.

[0062] Optionally, the predicted resource occupancy rate of each batch job on the target database may be pre-input by the target object, and the target object may be a developer of the target scheduling system.

[0063] Optionally, the predicted resource occupancy rate of each batch job on the target database may also be output by the target neural network model.

[0064] Optionally, after obtaining the current resource usage rate and predicted resource occupancy rate of the target database, the target processing system may determine whether the target database meets the preset condition based on the current resource usage rate and predicted resource occupancy rate of the target database. For example, based on the current resource usage rate and predicted resource occupancy rate of the target database, the target resource usage rate of the target database when executing multiple batch jobs is calculated, and then the target resource usage rate is compared with the preset usage rate to determine whether the target database meets the preset condition. The preset condition may be that the concurrency threshold of the target database will not cause resource shortage after being adjusted to the reference threshold, that is, the target resource usage rate is less than or equal to the preset usage rate.

[0065] Optionally, when it is determined that the target database meets the preset condition, it is determined that the target database will not experience resource shortage after being adjusted to the reference threshold, so that the parameters of the target database can be adjusted according to the reference threshold.

[0066] In some embodiments, in the process of judging whether the target database meets the preset conditions according to the current resource usage rate and the predicted resource occupancy rate of the target database, the target scheduling system may first judge whether the reference threshold is greater than the current threshold of the concurrency number of the target database. If the reference threshold is greater than the current threshold of the concurrency number of the target database, then judging whether the target database meets the preset conditions according to the current resource usage rate and the predicted resource occupancy rate of the target database. Conversely, if the reference threshold is less than or equal to the current threshold of the concurrency number of the target database, then directly adjusting the parameters of the target database according to the reference threshold.

[0067] It should be noted that, through the above process, after determining the reference threshold, the conditions for adjusting the target database based on the reference threshold are determined in combination with the resource usage of the target database, thereby effectively improving the rationality of adjusting the target database and further improving the job processing efficiency.

[0068] Optionally, in the batch job processing method provided in the embodiment of the present application, whether the target database meets the preset conditions is judged based on the current resource utilization rate and the predicted resource utilization rate of the target database, including: calculating the target resource utilization rate of the target database when executing multiple batch jobs based on the current resource utilization rate and the predicted resource utilization rate of the target database; when the target resource utilization rate is greater than the preset utilization rate, determining that the target database does not meet the preset conditions; when the target resource utilization rate is less than or equal to the preset utilization rate, determining that the target database meets the preset conditions.

[0069] Optionally, the current resource usage of the target database includes but is not limited to I / O usage and CPU usage.

[0070] Optionally, the predicted resource occupancy includes but is not limited to predicted I / O occupancy and predicted CPU occupancy.

[0071] Optionally, the preset usage includes but is not limited to a preset I / O usage and a preset CPU usage.

[0072] In some embodiments, the target scheduling system can calculate the sum of the I / O usage in the current resource usage and the predicted I / O usage in the predicted resource occupancy of multiple batch jobs to obtain the target I / O usage, and calculate the sum of the CPU usage in the current resource usage and the predicted CPU usage in the predicted resource occupancy of multiple batch jobs to obtain the target CPU usage.

[0073] In some embodiments, the target scheduling system may also sort the multiple batch jobs according to their job priorities (priorities output by the target neural network model), and then determine the first N batch jobs among the sorted multiple batch jobs as target batch jobs, where N is equal to the current idle concurrency of the target database, and the current idle concurrency is equal to the difference between the current threshold of the concurrency of the target database and the current concurrency. Afterwards, the target scheduling system may calculate the sum of the I / O usage in the current resource usage and the predicted I / O usage in the predicted resource occupancy of the multiple target batch jobs to obtain the target I / O usage, and calculate the sum of the CPU usage in the current resource usage and the predicted CPU usage in the predicted resource occupancy of the multiple target batch jobs to obtain the target CPU usage.

[0074] In some embodiments, the target resource utilization rate includes the above-mentioned target I / O utilization rate and target CPU utilization rate. Optionally, the target scheduling system may determine that the target resource utilization rate is greater than the preset utilization rate when the target I / O utilization rate is greater than the preset I / O utilization rate, or the target CPU utilization rate is greater than the preset CPU utilization rate, thereby determining that the target database does not meet the preset condition. The preset condition may be that the concurrency threshold of the target database will not cause resource shortage after being adjusted to the reference threshold. The target scheduling system may determine that the target resource utilization rate is less than or equal to the preset utilization rate when the target I / O utilization rate is less than or equal to the preset I / O utilization rate, and the target CPU utilization rate is less than or equal to the preset CPU utilization rate, thereby determining that the target resource utilization rate is greater than the preset utilization rate.

[0075] It should be noted that, through the above process, an accurate judgment of whether the target database meets the preset conditions is achieved, thereby facilitating further improving the rationality of parameter adjustment.

[0076] Optionally, in the batch job processing method provided in an embodiment of the present application, after determining whether the target database meets the preset conditions, the method also includes: generating early warning information when it is determined that the target database does not meet the preset conditions, wherein the early warning information indicates that there is a risk of exceeding the limit of resources of the target database.

[0077] Optionally, when it is determined that the target database does not meet the preset conditions, it is determined that the concurrency threshold of the target database may be resource-constrained after being adjusted to the reference threshold. Therefore, the target scheduling system can generate warning information and send the warning information to the target object for the target object to perform corresponding processing.

[0078] For example, Figure 3 is a schematic diagram of adjusting target database parameters according to an embodiment of the present application, such as Figure 3 As shown, after obtaining the reference threshold, the target database is judged whether it meets the preset conditions based on the current resource utilization rate and predicted resource occupancy rate of the target database. If it does, the parameters of the target database are adjusted according to the reference threshold. If it does not, early warning processing is performed.

[0079] It should be noted that, through the above process, timely warning of possible risks to the target database is achieved, thereby facilitating the improvement of the working stability of the target database and further improving the operation processing efficiency.

[0080] Optionally, in the batch job processing method provided in an embodiment of the present application, the job information and database information of multiple batch jobs are processed through a target neural network model to obtain the priorities corresponding to the multiple batch jobs and a reference threshold of the concurrency of the target database, including: processing the job information and database information of multiple batch jobs through a target neural network model to obtain the priorities corresponding to the multiple batch jobs, the reference threshold of the concurrency of the target database, and the predicted resource occupancy rate of each batch job for the target database.

[0081] In some embodiments, the database information of the target database includes the current threshold of the concurrency of the target database, the current concurrency, and the current resource utilization of the target database. The input data of the target neural network model includes: (1) job information, such as job name, job type, database where the job runs, current priority of the job, etc.; (2) database information, including at least the current threshold of the concurrency of the target database, the current concurrency, current resource utilization of the target database, etc.; (3) system warning parameters, i.e., the warning threshold of the concurrency of the target database. When the input data is the aforementioned content, the output data of the target neural network model includes at least: (1) job name; (2) job priority of the job recommended for adjustment; (3) reference threshold of the concurrency of the target database recommended for adjustment; (4) IO and CPU usage resources that may be occupied by batch jobs, i.e., predicted resource occupancy.

[0082] For example, Figure 4 is a working schematic diagram of the target neural network model provided according to an embodiment of the present application, such as Figure 4 As shown, the job information and database information are input into the target neural network model to obtain the scheduling parameters to be adjusted. The scheduling parameters to be adjusted include the above-mentioned job priority, reference threshold, and predicted resource occupancy rate.

[0083] Optionally, the predicted resource occupancy rate output by the target neural network model can be used to calculate the target resource usage rate of the target database when executing multiple batch jobs.

[0084] In some embodiments, the target neural network model can be trained in the following manner: construct a training sample set, wherein the training samples in the training sample set include job information of a sample batch job, database information of a sample database associated with the sample batch job, and the true labels of the training samples include at least the priority corresponding to the sample batch job, the reference threshold of the concurrency of the sample database, and the resource occupancy rate of the sample database by the sample batch job; and train the initial neural network model based on the constructed training sample set to obtain the target neural network model.

[0085] In some embodiments, the core idea of ​​the target neural network model is:

[0086] (1) If the target database has sufficient concurrency within the resource threshold, the concurrency number will not be adjusted, and the job priority will be increased, with important jobs running first.

[0087] (2) Within the resource threshold range, when the used concurrency of the target database is close to the existing total concurrency, the total concurrency is increased and the job priority is not adjusted;

[0088] (3) When resources are close to the threshold, the total concurrency of the scheduling system is reduced.

[0089] It should be noted that, through the above process, the predicted resource occupancy rate corresponding to each batch job is accurately determined, thereby improving the accuracy of database parameter adjustment and further improving the processing efficiency of the batch job.

[0090] Optionally, in the batch job processing method provided in an embodiment of the present application, before obtaining job information of multiple batch jobs to be executed, the method includes: receiving a data processing program sent by a target object, wherein the data processing program is used to process a data table in at least one database, and the at least one database includes a target database; identifying from the data processing program the data table targeted by the data processing program; and generating a batch job based on the data table targeted by the data processing program and the data processing program.

[0091] Optionally, the data processing program is pre-written by the developer (i.e., the target object), and the data processing program is used to process the data table in at least one database. Optionally, the developer can submit the specific data processing program in a specific directory, and the target scheduling system obtains the data processing program from the directory. The data processing program is used to perform operations such as data analysis, format conversion, or aggregation.

[0092] Optionally, after receiving the data processing program, the target scheduling system can automatically parse the program code and identify the data tables involved in the program. In this embodiment, the data table targeted by the identified data processing program is the data source table, and the target scheduling system can identify the data table from the program based on semantic analysis of the program code.

[0093] In some embodiments, after identifying the data table, the target scheduling system can generate a batch job for each data table, the batch job includes a job execution script, data table information of the data table, operation parameters of the job (such as priority, resource requirements, etc.), job name, etc., wherein the job execution script is determined from the data processing program. In the process of generating the batch job, the target scheduling system can analyze the dependency relationship between the scripts in the data processing program. If a batch job needs to rely on the job result of another batch job, then the corresponding dependency relationship will be set when generating the job.

[0094] For example, Figure 5 is a schematic diagram of generating a batch job according to an embodiment of the present application, such as Figure 5As shown, after identifying the data table, the target scheduling system can determine the job execution script corresponding to the data table from the data processing program, and then determine whether there is a batch job including the job execution script among the multiple batch jobs that have been constructed (that is, determine whether the job is newly added). If not, generate the batch job corresponding to the data table, and determine the corresponding dependency during the generation process. If it exists, determine that there is no need to add a new batch job, and determine the target dependency of the batch job based on the data processing program. If the current dependency of the batch job is different from the target dependency, adjust the current dependency of the batch job based on the target dependency. If the current dependency of the batch job is the same as the target dependency, determine that the batch job corresponding to the data table already exists in the target scheduling system.

[0095] It should be noted that, through the above process, automatic generation of batch jobs is achieved, thereby effectively reducing labor costs.

[0096] Optionally, in the batch job processing method provided in the embodiment of the present application, a batch job is generated based on the data table and the data processing program targeted by the data processing program, including: generating an initial batch job based on the data table and the data processing program targeted by the data processing program; sending the initial batch job to the target object; and upon receiving a job adjustment instruction from the target object, adjusting the initial batch job to obtain a batch job.

[0097] In some embodiments, the jobs generated by the target scheduling system according to the data table and the data processing program can be determined as the initial batch jobs, and then the initial batch jobs are manually adjusted to obtain the batch jobs.

[0098] For example, Figure 5 As shown, after the target scheduling system pre-generates batch jobs, the pre-generated batch jobs are determined as initial batch jobs, and then the initial batch jobs are pre-verified according to the job verification program (e.g., verifying the job name, job running frequency, scripts used by the job, etc.). After the pre-verification passes, the initial batch job is sent to the developer (i.e., the target object). If the pre-verification fails, it is determined that the batch job generation has failed, and corresponding warning information can be generated.

[0099] Optional, such as Figure 5 As shown, after receiving the initial batch job, the developer reviews the initial batch job to determine whether the initial batch job needs to be adjusted. If adjustment is required, a job adjustment instruction is issued to the target scheduling system to adjust the initial batch job until the developer's needs are met, and the adjusted initial batch job is determined as the batch job of the final application. If adjustment is not required, the initial batch job can be directly determined as the batch job of the final application.

[0100] It should be noted that, through the above method, manual review of batch jobs is achieved, thereby effectively improving the reliability of generated batch jobs.

[0101] In some embodiments, the target scheduling system may include a database connection module, a scheduling data collection module, a scheduling algorithm decision module, and a scheduling algorithm adjustment and monitoring module. Figure 6 is a working schematic diagram of the target scheduling system provided according to an embodiment of the present application, such as Figure 6 As shown, the database connection module provides common components that adapt to each database connection, as well as common development work components such as data loading components, data processing components, etc., which facilitate developers to develop data processing programs. The scheduling data acquisition module is used to collect database information and job information. The scheduling algorithm decision module is based on the information obtained by the scheduling data acquisition module. The intelligent algorithm trained by machine learning (i.e., the target neural network model) derives the scheduling strategy of the input job (i.e., the job priority and the reference threshold of the database). The scheduling algorithm adjustment and monitoring module is used to adjust the scheduling system parameters according to the results of the scheduling algorithm decision, that is, adjust the priority and the concurrency threshold, so that after the adjustment, the batch job is executed through the database. Optionally, the target scheduling system can also include an automatic job maintenance module, which is used to generate batch jobs according to the data processing program.

[0102] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining the job priority and the concurrent number threshold of the database when processing batch jobs, realizes the technical effect of improving the processing efficiency of batch jobs, and solves the problem of setting a fixed priority for batch jobs and a fixed concurrent number threshold for the database when processing batch jobs in the related technology, resulting in low efficiency in batch job processing.

[0103] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0104] Example 2

[0105] The present application also provides a batch job processing device. It should be noted that the batch job processing device of the present application can be used to execute the batch job processing method provided by the present application. The batch job processing device provided by the present application is introduced below.

[0106] According to an embodiment of the present application, a device for implementing the above-mentioned batch job processing method is also provided, such as Figure 7 As shown, the device comprises:

[0107] The acquisition module 701 is used to acquire job information of multiple batch jobs to be executed, and acquire database information of a target database for the multiple batch jobs to be executed, wherein the multiple batch jobs are associated with the same target database, and the database information at least includes a current threshold value of the number of concurrent connections and a current number of concurrent connections of the target database;

[0108] The first processing module 702 is used to process the job information and database information of multiple batch jobs through the target neural network model to obtain the priorities corresponding to the multiple batch jobs and the reference threshold of the concurrency number of the target database;

[0109] The second processing module 703 is used to adjust the parameters of the target database according to the reference threshold, and control the adjusted target database to execute the multiple batch jobs based on the priorities corresponding to the multiple batch jobs.

[0110] In an embodiment of the present application, by obtaining the job information of multiple batch jobs to be executed and obtaining the database information of the target database of the multiple batch jobs to be executed, the job characteristics of the batch jobs to be executed and the real-time working conditions of the database used to execute the jobs are effectively obtained. By using the target neural network model to process the job information and the database information, the priorities corresponding to the multiple batch jobs and the reference threshold of the concurrency of the target database are obtained, and the job priority and the concurrency threshold of the database are dynamically determined according to the job characteristics and the real-time working conditions of the database. Therefore, when executing batch jobs according to the dynamically determined job priority and the concurrency threshold of the database, the operation between jobs can be better coordinated, resource conflicts can be reduced, the waiting time of subsequent jobs that depend on data processing results can be reduced, and the rationality of the use of target database resources can be improved to avoid resource surplus or performance bottlenecks, thereby effectively improving the processing efficiency of batch jobs.

[0111] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining the job priority and the concurrent number threshold of the database when processing batch jobs, realizes the technical effect of improving the processing efficiency of batch jobs, and solves the problem of setting a fixed priority for batch jobs and a fixed concurrent number threshold for the database when processing batch jobs in the related technology, resulting in low efficiency in batch job processing.

[0112] Optionally, in the batch job processing device provided in the embodiment of the present application, the second processing module also includes: a determination submodule, used to determine the current resource utilization rate of the target database from the database information; an acquisition submodule, used to obtain the predicted resource utilization rate of each batch job on the target database; a judgment submodule, used to judge whether the target database meets the preset conditions based on the current resource utilization rate and the predicted resource utilization rate of the target database; and a first processing submodule, used to adjust the parameters of the target database according to a reference threshold when it is determined that the target database meets the preset conditions.

[0113] Optionally, in the batch job processing device provided in the embodiment of the present application, the judgment submodule also includes: a calculation unit, used to calculate the target resource utilization rate of the target database when executing multiple batch jobs based on the current resource utilization rate and the predicted resource occupancy rate of the target database; a first determination unit, used to determine that the target database does not meet the preset conditions when the target resource utilization rate is greater than the preset utilization rate; and a second determination unit, used to determine that the target database meets the preset conditions when the target resource utilization rate is less than or equal to the preset utilization rate.

[0114] Optionally, in the batch job processing device provided in the embodiment of the present application, the batch job processing device also includes: a first generation module, used to generate warning information when it is determined that the target database does not meet preset conditions, wherein the warning information indicates that there is a risk of exceeding the limit of resources in the target database.

[0115] Optionally, in the batch job processing device provided in the embodiment of the present application, the first processing module also includes: a second processing sub-module, which is used to process the job information and database information of multiple batch jobs through the target neural network model to obtain the priorities corresponding to the multiple batch jobs, the reference threshold of the concurrency number of the target database, and the predicted resource occupancy rate of each batch job for the target database.

[0116] Optionally, in the batch job processing device provided in the embodiment of the present application, the batch job processing device also includes: a receiving module, used to receive a data processing program sent by a target object, wherein the data processing program is used to process a data table in at least one database, and the at least one database includes a target database; an identification module, used to identify from the data processing program the data table targeted by the data processing program; and a second generation module, used to generate batch jobs based on the data table targeted by the data processing program and the data processing program.

[0117] Optionally, in the batch job processing device provided in the embodiment of the present application, the second generation module also includes: a generation sub-module, used to generate an initial batch job based on the data table and data processing program targeted by the data processing program; a sending sub-module, used to send the initial batch job to the target object; and a third processing sub-module, used to adjust the initial batch job to obtain a batch job when receiving a job adjustment instruction from the target object.

[0118] It should be noted that the acquisition module 701, the first processing module 702, and the second processing module 703 correspond to steps S201 to S203 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0119] Example 3

[0120] An embodiment of the present application may provide an electronic device, Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device may include: one or more ( Figure 8 (only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0121] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the job information of multiple batch jobs to be executed, and obtain the database information of the target database for executing the multiple batch jobs, wherein the multiple batch jobs are associated with the same target database, and the database information at least includes the current threshold value and the current concurrency number of the target database; process the job information and database information of the multiple batch jobs through the target neural network model to obtain the priorities corresponding to the multiple batch jobs and the reference threshold value of the concurrency number of the target database; adjust the parameters of the target database according to the reference threshold value, and control the adjusted target database to execute the multiple batch jobs based on the priorities corresponding to the multiple batch jobs.

[0123] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the current resource utilization rate of the target database from the database information; obtain the predicted resource utilization rate of each batch job on the target database; determine whether the target database meets the preset conditions based on the current resource utilization rate and the predicted resource utilization rate of the target database; and when it is determined that the target database meets the preset conditions, adjust the parameters of the target database based on the reference threshold.

[0124] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: based on the current resource utilization rate and predicted resource occupancy rate of the target database, calculate the target resource utilization rate of the target database when executing multiple batch jobs; when the target resource utilization rate is greater than the preset utilization rate, determine that the target database does not meet the preset conditions; when the target resource utilization rate is less than or equal to the preset utilization rate, determine that the target database meets the preset conditions.

[0125] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: after determining whether the target database meets the preset conditions, if it is determined that the target database does not meet the preset conditions, generate warning information, wherein the warning information indicates that there is a risk of exceeding the limit of resources in the target database.

[0126] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: process the job information and database information of multiple batch jobs through the target neural network model to obtain the priorities corresponding to the multiple batch jobs, the reference threshold of the concurrency number of the target database, and the predicted resource occupancy rate of each batch job for the target database.

[0127] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: before obtaining the job information of multiple batch jobs to be executed, receive a data processing program sent by the target object, wherein the data processing program is used to process a data table in at least one database, and at least one database includes a target database; identify the data table targeted by the data processing program from the data processing program; and generate batch jobs based on the data table targeted by the data processing program and the data processing program.

[0128] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: generate an initial batch job based on the data table and data processing program targeted by the data processing program; send the initial batch job to the target object; and adjust the initial batch job to obtain a batch job when receiving a job adjustment instruction from the target object.

[0129] In an embodiment of the present application, by obtaining the job information of multiple batch jobs to be executed and obtaining the database information of the target database of the multiple batch jobs to be executed, the job characteristics of the batch jobs to be executed and the real-time working conditions of the database used to execute the jobs are effectively obtained. By using the target neural network model to process the job information and the database information, the priorities corresponding to the multiple batch jobs and the reference threshold of the concurrency of the target database are obtained, and the job priority and the concurrency threshold of the database are dynamically determined according to the job characteristics and the real-time working conditions of the database. Therefore, when executing batch jobs according to the dynamically determined job priority and the concurrency threshold of the database, the operation between jobs can be better coordinated, resource conflicts can be reduced, the waiting time of subsequent jobs that depend on data processing results can be reduced, and the rationality of the use of target database resources can be improved to avoid resource surplus or performance bottlenecks, thereby effectively improving the processing efficiency of batch jobs.

[0130] It can be seen that the solution provided by the present application achieves the purpose of dynamically determining the job priority and the concurrent number threshold of the database when processing batch jobs, realizes the technical effect of improving the processing efficiency of batch jobs, and solves the problem of setting a fixed priority for batch jobs and a fixed concurrent number threshold for the database when processing batch jobs in the related technology, resulting in low efficiency in batch job processing.

[0131] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 8The structure of the electronic device is not limited. Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations shown.

[0132] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0133] Example 4

[0134] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the batch job processing method provided in the first embodiment.

[0135] Optionally, in this embodiment, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0136] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program of the processing method steps of a batch job.

[0137] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0138] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0143] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A batch job processing method, characterized in that: include: Obtaining job information of a plurality of batch jobs to be executed, and obtaining database information of a target database for executing the plurality of batch jobs, wherein the plurality of batch jobs are associated with the same target database, and the database information at least includes a current threshold value and a current concurrency number of the target database; Processing the job information of the plurality of batch jobs and the database information through a target neural network model to obtain priorities corresponding to the plurality of batch jobs and a reference threshold value of the concurrency number of the target database; The parameters of the target database are adjusted according to the reference threshold, and the adjusted target database is controlled to execute the multiple batch jobs based on the priorities corresponding to the multiple batch jobs.

2. The method according to claim 1, characterized in that Adjusting the parameters of the target database according to the reference threshold includes: Determining a current resource usage rate of the target database from the database information; Obtaining predicted resource occupancy rates of each batch job on the target database; Determining whether the target database meets a preset condition according to the current resource usage rate of the target database and the predicted resource occupancy rate; When it is determined that the target database satisfies the preset condition, the parameters of the target database are adjusted according to the reference threshold.

3. The method according to claim 2, characterized in that Judging whether the target database meets a preset condition according to the current resource usage rate of the target database and the predicted resource usage rate includes: Calculating a target resource utilization rate of the target database when executing the plurality of batch jobs according to the current resource utilization rate of the target database and the predicted resource utilization rate; When the target resource usage rate is greater than a preset usage rate, determining that the target database does not meet the preset condition; When the target resource usage rate is less than or equal to the preset usage rate, it is determined that the target database meets the preset condition.

4. The method according to claim 2, characterized in that: After determining whether the target database meets the preset conditions, the method further includes: When it is determined that the target database does not meet the preset condition, a warning message is generated, wherein the warning message indicates that there is a risk of exceeding the limit of resources of the target database.

5. The method according to claim 2, characterized in that: Processing the job information of the plurality of batch jobs and the database information through a target neural network model to obtain priorities corresponding to the plurality of batch jobs and a reference threshold value of the number of concurrent operations of the target database includes: The job information of the multiple batch jobs and the database information are processed through the target neural network model to obtain the priorities corresponding to the multiple batch jobs, the reference threshold of the concurrency number of the target database, and the predicted resource occupancy rate of each batch job on the target database.

6. The method according to claim 1, characterized in that Before obtaining job information of a plurality of batch jobs to be executed, the method includes: Receiving a data processing program sent by a target object, wherein the data processing program is used to process a data table in at least one database, wherein the at least one database includes the target database; identifying, from the data processing program, a data table targeted by the data processing program; A batch job is generated based on the data table targeted by the data processing program and the data processing program.

7. The method according to claim 6, characterized in that Generating a batch job according to the data table targeted by the data processing program and the data processing program, including: generating an initial batch job according to the data table targeted by the data processing program and the data processing program; Sending the initial batch job to the target object; When the job adjustment instruction of the target object is received, the initial batch job is adjusted to obtain the batch job.

8. A batch job processing device, characterized in that: include: an acquisition module, configured to acquire job information of a plurality of batch jobs to be executed, and acquire database information of a target database for executing the plurality of batch jobs, wherein the plurality of batch jobs are associated with the same target database, and the database information at least includes a current threshold value and a current concurrency number of the target database; A first processing module is used to process the job information of the plurality of batch jobs and the database information through a target neural network model to obtain the priorities corresponding to the plurality of batch jobs and a reference threshold value of the concurrency number of the target database; The second processing module is configured to adjust the parameters of the target database according to the reference threshold, and control the adjusted target database to execute the multiple batch jobs based on the priorities corresponding to the multiple batch jobs.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the batch job processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the batch job processing method described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the batch job processing method described in any one of claims 1 to 7 are implemented.

Images