Batch task processing method, system and equipment and storage medium

Abstract

The invention provides a batch task processing method, system and device and a storage medium, and relates to the technical field of computer application. The batch task processing method comprises the following steps: acquiring to-be-processed batch processing data and batch processing data volume; dynamically determining the number of application services for processing the batch processing data according to the batch processing data volume; determining a data distribution amount sent to each application service according to the batch processing data amount and the number of the application services; and each application service obtains to-be-processed data according to the corresponding data distribution amount. By adopting the method and the device, the quantity of the application services for batch processing and the data distribution quantity sent to each application service can be dynamically determined according to the to-be-processed batch processing data and the batch processing data quantity, so that the data processing tasks are reasonably distributed, the data processing efficiency is ensured, and resource waste is avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and more specifically, to a batch task processing method, system, device and storage medium. Background Art

[0002] As the scope of the system business continues to expand, the amount of data processing has increased dramatically, which has posed severe challenges to the timeliness and resource utilization efficiency of data batch processing. At present, it mainly relies on the online deployment of a single independent batch instance (application service that processes batch data / tasks) and pre-allocates static resources to the batch instance to ensure that the batch tasks can be executed quickly. However, this data batch processing method has the following problems:

[0003] 1. The characteristics of batch tasks determine that they do not consume a lot of resources during non-running time. However, in the static resource allocation mode, in order to ensure peak processing capacity, sufficient resources must be reserved for each batch application, even if these resources are idle most of the time, resulting in huge resource waste.

[0004] 2. The single-instance deployment batch processing method will cause the batch processing time to be too long when the data surges, affecting data accuracy and security. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned prior art or at least part of the problems, the embodiments of the present invention provide a batch task processing method, system, device, storage medium and computer program product, which dynamically determine the number of application services used for batch processing and the data distribution volume sent to each application service according to the batch data to be processed and the batch data volume, and can reasonably allocate data processing tasks, ensure data processing efficiency, and avoid waste of resources.

[0006] According to a first aspect of the present invention, an embodiment of the present invention provides a batch task processing method, which includes: obtaining batch data to be processed and the amount of batch data; dynamically determining the number of application services that process the batch data based on the amount of batch data; determining the data distribution amount to be sent to each application service based on the amount of batch data and the number of application services; each application service obtains the data to be processed according to its corresponding data distribution amount.

[0007] According to the above-mentioned embodiment of the present invention, by monitoring the batch data to be processed and the amount of batch data, and determining the number of application services used to process the batch data and the amount of data distribution sent to each application service, it is possible to reasonably allocate data processing tasks according to the amount of data to be processed, ensure data processing efficiency, improve data accuracy and security, and at the same time achieve elastic scaling scheduling based on the tasks to be processed to avoid waste of resources.

[0008] In some embodiments of the present invention, obtaining the batch data to be processed and the amount of batch data includes: obtaining the file to be processed and parsing the file to be processed to obtain standard data in a preset format; generating the batch data to be processed according to the auto-increment ID and the standard data; and determining the amount of batch data according to the auto-increment ID.

[0009] In some embodiments of the present invention, determining the data distribution amount to be sent to each application service based on the batch data amount and the number of application services includes: obtaining the number of preset batch data items; calculating the auto-increment ID range of the batch data that needs to be processed by each application service based on the maximum auto-increment ID and the minimum auto-increment ID corresponding to the batch data to be processed, and the number of preset batch data items.

[0010] In some implementations of the present invention, the batch task processing method further includes: deleting the data to be processed from the batch data to be processed after the application service completes processing of the data to be processed; and reclaiming resources occupied by the application service.

[0011] According to the above-mentioned implementation mode of the present invention, by dynamically recovering the resources occupied by the application service after the batch running is completed, the waste of resources is further avoided.

[0012] According to the second aspect of the present invention, an embodiment of the present invention provides a batch task processing system, which includes: a data monitoring module, used to obtain batch data to be processed and the batch data volume; an application service calculation module, used to dynamically determine the number of application services that process the batch data according to the batch data volume; a distribution volume determination module, used to determine the data distribution volume sent to each application service according to the batch data volume and the number of application services; a data acquisition module, used to enable each application service to obtain the data to be processed according to its corresponding data distribution volume.

[0013] According to the above-mentioned embodiment of the present invention, by monitoring the batch data to be processed and the amount of batch data, and determining the number of application services used to process the batch data and the amount of data distribution sent to each application service, it is possible to reasonably allocate data processing tasks according to the amount of data to be processed, ensure data processing efficiency, improve data accuracy and security, and at the same time achieve elastic scaling scheduling based on the tasks to be processed to avoid waste of resources.

[0014] In some embodiments of the present invention, obtaining the batch data to be processed and the amount of batch data includes: obtaining the file to be processed and parsing the file to be processed to obtain standard data in a preset format; generating the batch data to be processed according to the auto-increment ID and the standard data; and determining the amount of batch data according to the auto-increment ID.

[0015] In some embodiments of the present invention, determining the data distribution amount to be sent to each application service based on the batch data amount and the number of application services includes: obtaining the number of preset batch data items; calculating the auto-increment ID range of the batch data that needs to be processed by each application service based on the maximum auto-increment ID and the minimum auto-increment ID corresponding to the batch data to be processed, and the number of preset batch data items.

[0016] In some embodiments of the present invention, the batch task processing system also includes a resource recovery module for performing the following operations: deleting the data to be processed from the batch data to be processed after the application service completes processing of the data to be processed; and recovering the resources occupied by the application service.

[0017] According to the above-mentioned implementation mode of the present invention, by dynamically recovering the resources occupied by the application service after the batch running is completed, the waste of resources is further avoided.

[0018] According to the third aspect of the present invention, an embodiment of the present invention provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor, the computer performs the following operations: the operations include the steps included in the batch task processing method described in any one of the above embodiments.

[0019] According to the fourth aspect of the present invention, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory is used to store one or more computer-readable instructions, wherein the one or more computer-readable instructions, when executed by the processor, can implement the batch task processing method described in any one of the above embodiments.

[0020] According to a fifth aspect of the present invention, an embodiment of the present invention provides a computer program product including a computer program, and when the computer program is executed by a processor, the batch task processing method described in any one of the above embodiments is implemented.

[0021] From the above, it can be seen that the batch task processing method, system, device, storage medium and computer program product provided in the embodiments of the present invention can reasonably allocate data processing tasks according to the amount of data to be processed, ensure data processing efficiency, improve data accuracy and security, and at the same time achieve elastic scaling scheduling based on the tasks to be processed to avoid waste of resources by monitoring the batch data to be processed and the amount of batch data, and determining the number of application services used to process the batch data and the amount of data distribution sent to each application service. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1is a flow chart of a batch task processing method according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flow chart of a batch task processing method according to Embodiment 2 of the present invention;

[0024] Figure 3 2 is a schematic diagram of the architecture of a batch task processing system according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0025] The various aspects of the present invention are described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, well-known modules, units and their connections, links, communications or operations are not shown or described in detail. In addition, the described features, architectures or functions can be combined in any way in one or more embodiments. It should be understood by those skilled in the art that the various embodiments described below are only used for illustration and are not intended to limit the scope of protection of the present invention. It can also be easily understood that the modules or units or processing methods in each embodiment described herein and shown in the accompanying drawings can be combined and designed in various different configurations.

[0026] The following is a brief description of the terms used in this article.

[0027] Master node: the main node or coordination node.

[0028] Batch running: At a specified time point or time period, the system automatically triggers a series of pre-set programs or tasks to centrally process large amounts of data.

[0029] Batch ID: The unique identifier obtained for each batch.

[0030] Auto-increment ID: ID generated by MySQL auto-increment primary key.

[0031] Redis: An open source in-memory data structure storage system that can be used as a cache to improve data access speed and system performance.

[0032] API: Application Programming Interface, application programming interface.

[0033] MySQL: Open source relational database management system.

[0034] MQ: Message Queue, message queue.

[0035] k8s: An open source container orchestration platform for automating the deployment, scaling, and management of containerized applications.

[0036] [Example 1]

[0037] Figure 1 4 is a flow chart of a batch task processing method according to Embodiment 1 of the present invention.

[0038] like Figure 1 As shown, in Embodiment 1 of the present invention, the batch task processing method may include: step S11, step S12, step S13 and step S14, and the above steps are described in detail below.

[0039] In step S11, batch data to be processed and the batch data volume are obtained.

[0040] In some embodiments, obtaining the batch data to be processed and the amount of batch data includes: obtaining the file to be processed and parsing the file to be processed to obtain standard data in a preset format; generating the batch data to be processed according to the auto-increment ID and the standard data; and determining the amount of batch data according to the auto-increment ID.

[0041] In step S12, the number of application services for processing the batch data is dynamically determined according to the amount of the batch data.

[0042] In some embodiments, the master node senses the amount of batch data to be run, automatically calculates the instances required for the batch run (application services that process batch data / tasks), and then starts the corresponding instances to run the batch. For example, before running the batch, the master node dynamically determines the number of batch processing instances required by monitoring the amount of batch data, including: determining the required number of batch processing instances (number of application services) by calculating the ratio of the total number of batch processing data to the upper limit of single-machine processing (i.e., the number of preset batch processing data). The required number of batch processing instances is obtained through the above calculation, which can ensure that the data processed by each instance is uniform.

[0043] In a further embodiment, the number of required batch processing instances can be dynamically adjusted based on the amount of historical data, current load conditions, and predicted demand. For example, the theoretical number of batch processing instances for the current channel batch is calculated by the following steps: (1) Collect historical data to analyze the amount of data that needs to be processed every day and the data processed by a single application in each time period (such as every hour); (2) Collect application resource usage such as network bandwidth, memory usage, and garbage collection; (3) Calculate the theoretical number of batch processing instances for the current channel based on business needs.

[0044] In step S13, the data distribution amount sent to each application service is determined according to the batch data amount and the number of application services.

[0045] In some embodiments, determining the data distribution amount to be sent to each application service based on the batch data amount and the number of application services includes: obtaining the number of preset batch data items; calculating the auto-increment ID range of the batch data that needs to be processed by each application service based on the maximum auto-increment ID and the minimum auto-increment ID corresponding to the batch data to be processed, and the number of preset batch data items.

[0046] In this embodiment, data distribution is achieved by a hash-based data sharding method. For example, the number of shards is determined, the hash value is calculated and modulo operation is performed, and the corresponding data shards are obtained. In this way, some instances can be prevented from being overloaded while other instances are idle, thereby maximizing the use of instances for batch running.

[0047] In step S14, each application service obtains the data to be processed according to its corresponding data distribution volume.

[0048] In this embodiment, the application service uses message queue (MQ) consumption tasks and Redis cache technology to process data in slices during batch processing. Specifically, first, when the client synchronizes data, the master node assigns primary key information to each synchronized data, and puts the complete information into Redis, where the key of Redis is composed of a self-incrementing primary key and a unique identifier of this data. Secondly, after the client data is synchronized, the master node is informed. After the master node determines the data to be processed for each batch of each instance, it sends the self-incrementing ID range of the processed data through MQ. When the batch instance listens to the message, it obtains the data range processed by itself, and then obtains the corresponding data from Redis to start the real batch running. Among them, the hash modulus operation is performed according to the primary key and the unique identifier of the synchronized data, and then the data is assigned to a specific Redis slice. The message queue can ensure that the tasks are distributed to each instance in order, and the Redis slice can improve the read and write efficiency and parallel processing capabilities of the data, thereby significantly reducing the batch processing time.

[0049] After the Master node senses that the data batch is finished, it sends the data of each batch instance through MQ; after the batch instance listens to the MQ message, it can know the data range of the batch processing, and then obtain the complete data information from Redis based on the auto-increment ID and unique identifier. In this way, multiple instances can run batches together to improve the batch processing speed.

[0050] In a further embodiment, the processing progress of the batch data to be processed and the system load are monitored in real time, and the number of application services and the data distribution volume sent to each application service are adjusted according to the processing progress and system load. By monitoring the processing progress and system load of the task in real time and dynamically adjusting the task distribution volume and the number of instances, data processing tasks can be reasonably allocated to ensure data processing efficiency and avoid resource waste.

[0051] By adopting the above-mentioned batch task processing method of Example 1 of the present invention, by monitoring the batch data to be processed and the amount of batch data, and determining the number of application services used to process the batch data and the data distribution amount sent to each application service, it is possible to reasonably allocate data processing tasks according to the amount of data to be processed, ensure data processing efficiency, improve data accuracy and security, and at the same time achieve elastic scaling scheduling based on the tasks to be processed to avoid waste of resources.

[0052] In a further embodiment, the batch task processing method further includes: when the application service completes processing of the data to be processed, deleting the data to be processed from the batch data to be processed; and recycling the resources occupied by the application service. By recycling the instance resources that are no longer needed in time after the batch task is completed, resource waste can be avoided, and the automatic expansion function can be further realized to perform dynamic resource management.

[0053] [Example 2]

[0054] Figure 2 1 is a flow chart of a batch task processing method according to Embodiment 2 of the present invention. Embodiment 2 of the present invention is applied to a single machine deployment (ie, only one application instance is deployed on the server).

[0055] like Figure 2 As shown, in Embodiment 2 of the present invention, the batch task processing method may include: step S21, step S22, step S23, step S24, step S25 and step S26, and the above steps are described in detail below.

[0056] In step S21, the client connects to the standard API interface to synchronize data. The specific implementation method is as follows:

[0057] 1.1 The client first obtains the batch ID through the standard API interface. The batch ID is the unique identifier of this batch task. After calling the interface to create a batch task, the global batch ID is returned. One batch task / batch task processing corresponds to one batch ID.

[0058] 1.2 The client prepares standard data and synchronizes it to the server. The standard data refers to converting the data in the file into standard data that meets our requirements. In some embodiments, the client prepares standard data and synchronizes it to the server includes: the client pulls the to-be-processed file from the lending channel, then parses the to-be-processed file, converts the content in the file into the standard data defined by us, and then synchronizes the standard data to the server through the API.

[0059] 1.3 After the client completes data synchronization, it sends the global batch ID to the server again and informs the server that data synchronization is complete.

[0060] In step S22, the server receives the data synchronized by the client, performs data verification on the synchronized data, and returns the global taskId. The data verification includes: calculating whether the data obtained from the channel is complete and whether the code value is accurate; the global taskId is the batch ID obtained according to the API interface in the above step S21.

[0061] In step S23, the server calculates the total amount of data to be processed (batch data volume). In this embodiment, the master node detects the amount of data synchronized from the client to the server through the global batch ID. Specifically, when the client informs the batch service after the data synchronization is completed, the batch service calculates the data of this batch according to the global batch ID.

[0062] In step S24, the master node dynamically calculates the required number of batch processing instances based on the total amount of data monitored, and starts the batch processing instance (running the batch instance). In this embodiment, the required number of batch processing instances (number of application services) is determined by calculating the ratio of the total amount of data to be processed (batch processing data volume) to the upper limit of single-machine processing (i.e., the number of preset batch processing data), wherein the upper limit of single-machine processing is calculated in the following way: single-machine processing volume = the amount of data processed per hour when the memory utilization rate reaches 90% multiplied by 10 hours.

[0063] In step S25, the data shards are notified to each batch instance and batch processing is performed.

[0064] In some implementations, when synchronizing data, the server uses MySQL's auto-increment primary key to assign primary key information to each synchronized data. The server calculates the auto-increment id interval of each batch instance's processed data based on the minimum auto-increment id, maximum auto-increment id, and number of processed records in each batch in the batch database, and informs each batch instance of its corresponding auto-increment id interval of processed data through MQ. In this way, it can be ensured that each instance can process the assigned data.

[0065] In this embodiment, the data distribution amount is calculated according to the total amount of data to be processed (batch data amount), including: calculating the data distribution amount according to the formula end = min (minId + batchSize, maxId), wherein minId is the minimum auto-increment id, batchSize is the fixed batch processing data (i.e., the upper limit of the data processed by the batch instance per single time, which is obtained according to the maximum single processing volume of the database), and maxId is the maximum auto-increment id, that is, the data processed by each instance per batch is the minimum value of "minimum auto-increment id + fixed batch processing data" compared with the maximum auto-increment id, and the calculated data distribution amount is the data range distributed to each batch service processing. In this way, the data to be processed can be evenly distributed to each batch instance.

[0066] When the batch running instance listens to the MQ message, it queries the database according to the auto-increment id interval / range to obtain the redis key value, which is calculated based on the auto-increment id and the unique identifier of the data, and then obtains the complete data to be synchronized from redis. The message queue can ensure that tasks are distributed in order, and the use of Redis can improve data reading and writing efficiency and parallel processing capabilities.

[0067] In a further embodiment, during the real-time calculation of data distribution volume, the system continuously monitors data distribution to ensure that data distribution volume can be dynamically allocated and adjusted between different batch instances to maintain the overall load balance of the system. In an optional embodiment, k8s automatic expansion and contraction is used to adjust the number of instances, and k8s service and ingress are used to achieve service load balancing.

[0068] In step S26, the batch is finished and resources are recycled. The specific implementation method is as follows:

[0069] 2.1 The server detects the running status and task completion status of all batch instances to ensure that all instances have processed the data assigned to them.

[0070] 2.2 Record the end status of the batch task, and determine whether to release redundant batch processing instances based on the system load. For example, when the system load allows (for example, the CPU usage and / or memory usage is within the set minimum threshold of 30% to the set maximum threshold of 90%), release redundant instances to save resources. In some embodiments, after the batch is completed, the CPU usage and memory usage of the service are monitored by k8s, and the instance is released when the CPU usage and / or memory usage is less than the set threshold (for example, 30%).

[0071] 2.3 After confirming that all batch running instances have completed their tasks, refresh or clear caches such as Redis and process unprocessed tasks in the message queue. This can avoid backlogs and confusion, and facilitate the preparation and execution of the next round of batch running tasks.

[0072] 2.4 Update the system status and logs to record the end time and running status of this batch task for subsequent analysis and monitoring.

[0073] By recycling and cleaning up resources after the batch task is completed, the occupied computing resources and memory can be released.

[0074] By adopting the above-mentioned batch task processing method of Example 2 of the present invention, by monitoring the batch data to be processed and the amount of batch data, and determining the number of running batch instances used to process the batch data and the data distribution amount sent to each running batch instance, it is possible to reasonably allocate data processing tasks according to the amount of data to be processed, ensure data processing efficiency, improve data accuracy and security, and at the same time realize elastic scaling scheduling based on the tasks to be processed to avoid waste of resources.

[0075] In a further implementation, the client obtains the batch running result of the server through taskId.

[0076] In this embodiment, the batch task processing method described in Example 2 of the present invention is implemented based on the Java version 1.8 or above programming language and using k8s, and the data types processed in Example 2 of the present invention include but are not limited to json format.

[0077] [Example 3]

[0078] Embodiment 3 of the present invention provides a batch task processing method applied to distributed deployment (i.e., deploying multiple application instances on a server), the batch task processing method comprising the following steps:

[0079] Step 1: The client connects to the standard API interface and performs the following steps:

[0080] 1.2 The client first obtains the batch ID through the standard API interface. The batch ID is the unique identifier of this batch task.

[0081] 1.2 The client prepares standard data and synchronizes it to the server.

[0082] 1.3 After the client completes data synchronization, it notifies the server to ensure that data synchronization is complete.

[0083] Step 2: After receiving the data synchronized by the client, the server executes the following steps:

[0084] 2.1 Return the global batch ID to the client.

[0085] 2.2 The master node dynamically calculates the required number of batch processing instances by monitoring the amount of data to be processed, and starts the batch processing instances (running batch instances).

[0086] 2.3 Notify each batch instance of the data shards to be processed to ensure that each instance can process the assigned data.

[0087] Step 3: Run batch processing, which includes the following steps:

[0088] 3.1 Calculate the data distribution amount based on the amount of data to be processed and distribute it evenly to each batch instance.

[0089] 3.2 Use message queue (MQ) consumption tasks and Redis cache technology to shard and process data.

[0090] 3.3 During the real-time calculation of the number of batch instances and the amount of data distributed, the system continuously monitors the data distribution to ensure that the data volume can be dynamically allocated and adjusted between different batch instances to maintain the overall load balance of the system.

[0091] Step 4: The batch run is completed and resources are recycled, which specifically includes the following steps:

[0092] 4.1 The server detects the running status and task completion status of all batch instances to ensure that all instances have processed the data assigned to them.

[0093] 4.2 Record the end status of the batch task and decide whether to release the redundant batch processing instances based on the system load. If the system load allows, the redundant instances may be released to save resources.

[0094] 4.3 After confirming that the tasks of all batch running instances are completed, refresh or clear caches such as Redis, process unprocessed tasks in the message queue, avoid backlogs and confusion, and facilitate the preparation and execution of the next round of batch running tasks.

[0095] 4.4 Update the system status and logs to record the end time and running status of this batch task for subsequent analysis and monitoring.

[0096] 4.5 After the batch task is completed, the system recycles and cleans up resources to release the occupied computing resources and memory.

[0097] Step 5: The client obtains the batch running results of the server through taskId.

[0098] The batch task processing method of Embodiment 3 of the present invention is adopted. By monitoring the batch data to be processed and the amount of batch data, and determining the number of batch instances used to process the batch data and the amount of data distribution sent to each batch instance, data processing tasks can be reasonably allocated according to the amount of data to be processed, data processing efficiency can be ensured, data accuracy and security can be improved, and elastic scaling scheduling based on the tasks to be processed can be achieved to avoid resource waste. At the same time, by deploying multiple application instances and master nodes on the service, the high availability of the system can be improved, the fault tolerance can be improved, and the scalability can be improved.

[0099] [Example 4]

[0100] Figure 3 2 is a schematic diagram of the architecture of a batch task processing system according to Embodiment 4 of the present invention.

[0101] like Figure 3 As shown, the batch task processing system includes: a data monitoring module 310 , an application service calculation module 320 , a distribution volume determination module 330 , a data acquisition module 340 , and a resource recovery module 350 .

[0102] The data monitoring module 310 is used to obtain the batch data to be processed and the batch data volume. In some embodiments, the obtaining of the batch data to be processed and the batch data volume includes: obtaining the file to be processed and parsing the file to be processed to obtain standard data in a preset format; generating the batch data to be processed according to the self-incrementing ID and the standard data; and determining the batch data volume according to the self-incrementing ID.

[0103] The application service calculation module 320 is used to dynamically determine the number of application services that process the batch data according to the amount of the batch data.

[0104] The distribution volume determination module 330 is used to determine the data distribution volume sent to each application service according to the batch data volume and the number of application services. In some embodiments, determining the data distribution volume sent to each application service according to the batch data volume and the number of application services includes: obtaining the number of preset batch data; calculating the range of the auto-increment ID of the batch data that each application service needs to process according to the maximum auto-increment ID and the minimum auto-increment ID corresponding to the batch data to be processed and the number of preset batch data.

[0105] The data acquisition module 340 is used to enable each application service to acquire the data to be processed according to its corresponding data distribution volume.

[0106] The resource recovery module 350 is used to perform the following operations: when the application service completes processing of the data to be processed, delete the data to be processed from the batch data to be processed; and recover the resources occupied by the application service.

[0107] The batch task processing system of Example 4 of the present invention can reasonably allocate data processing tasks according to the amount of data to be processed, ensure data processing efficiency, improve data accuracy and security, and realize elastic scaling scheduling based on tasks to be processed to avoid resource waste by monitoring the batch data to be processed and the amount of batch data, and determining the number of application services used to process the batch data and the amount of data distribution sent to each application service. In addition, by timely recovering instance resources that are no longer needed after the batch task is completed, resource waste can be avoided, and automatic expansion function can be further realized to perform dynamic resource management.

[0108] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by combining software with a hardware platform. Based on such an understanding, all or part of the contribution of the technical solution of the present invention to the background technology can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation modes of the present invention or some parts of the implementation modes.

[0109] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium on which computer-readable instructions or programs are stored. When the computer-readable instructions or programs are executed by a processor, the computer performs the following operations: the operations include the steps included in the batch task processing method described in any of the above embodiments, which are not repeated here. The storage medium may include, for example, a CD, a hard disk, a floppy disk, a flash memory, a magnetic tape, etc.

[0110] In addition, an embodiment of the present invention further provides a computer device including a memory and a processor, wherein the memory is used to store one or more computer-readable instructions or programs, wherein the one or more computer-readable instructions or programs can implement the batch task processing method described in any of the above embodiments when executed by the processor. The computer device can be, for example, a server, a desktop computer, a notebook computer, a tablet computer, etc.

[0111] The present invention also provides a computer program product including a computer program, wherein the computer program includes program code for executing the batch task processing method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the batch task processing method provided by the embodiment of the present disclosure.

[0112] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention. Therefore, the protection scope of the present invention shall be subject to the claims.

Claims

1. A batch task processing method, characterized in that: The batch task processing method comprises: Get the batch data to be processed and the batch data volume; Dynamically determine the number of application services that process the batch data according to the amount of the batch data; Determining a data distribution amount to be sent to each application service according to the batch data amount and the number of the application services; Each application service obtains the data to be processed according to its corresponding data distribution volume.

2. The batch task processing method according to claim 1, characterized in that: The obtaining of the batch data to be processed and the batch data amount comprises: Acquire a file to be processed and parse the file to be processed to obtain standard data in a preset format; Generate the batch data to be processed according to the self-increment ID and the standard data; The batch processing data amount is determined according to the self-incrementing ID.

3. The batch task processing method according to claim 2, characterized in that: Determining the data distribution amount sent to each application service according to the batch processing data amount and the number of the application services includes: Get the number of preset batch processing data; The range of the auto-increment IDs of the batch data to be processed by each application service is calculated according to the maximum auto-increment ID and the minimum auto-increment ID corresponding to the batch data to be processed and the number of preset batch data.

4. The batch task processing method according to claim 3, characterized in that: The batch task processing method further includes: When the application service completes processing of the data to be processed, the data to be processed is deleted from the batch data to be processed; and Reclaim the resources occupied by the application service.

5. A batch task processing system, characterized in that: The batch task processing system comprises: A data monitoring module is used to obtain the batch data to be processed and the batch data volume; An application service calculation module, used for dynamically determining the number of application services that process the batch data according to the amount of the batch data; A distribution volume determination module, configured to determine a data distribution volume sent to each application service according to the batch processing data volume and the number of application services; The data acquisition module is used to enable each application service to acquire the data to be processed according to its corresponding data distribution volume.

6. The batch task processing system according to claim 5, characterized in that: The obtaining of the batch data to be processed and the batch data amount comprises: Acquire a file to be processed and parse the file to be processed to obtain standard data in a preset format; Generate the batch data to be processed according to the self-increment ID and the standard data; Determine the batch processing data amount according to the self-incrementing ID; Determining the data distribution amount sent to each application service according to the batch processing data amount and the number of the application services includes: Get the number of preset batch processing data; The range of the auto-increment IDs of the batch data to be processed by each application service is calculated according to the maximum auto-increment ID and the minimum auto-increment ID corresponding to the batch data to be processed and the number of preset batch data.

7. The batch task processing system according to claim 6, characterized in that: The batch task processing system also includes a resource recovery module, which is used to perform the following operations: When the application service completes processing of the data to be processed, the data to be processed is deleted from the batch data to be processed; and Reclaim the resources occupied by the application service.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that: The computer-readable instructions are executed by a processor to implement the batch task processing method according to any one of claims 1 to 4.

9. A computer device comprising a memory and a processor, The memory stores computer-readable instructions, characterized in that: The processor executes the computer-readable instructions to implement the batch task processing method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by the processor, the batch task processing method according to any one of claims 1 to 4 is implemented.

Images