Batch task processing method and device and computer program product

Abstract

The invention provides a batch task processing method and device and a computer program product. The method comprises the steps that to-be-processed batch tasks are acquired; under the condition that the data processing operation corresponding to the to-be-processed batch task is a multi-step operation, the to-be-processed batch task is executed in a fragmentation mode, the multi-step operation comprises data input, data processing and data output, and the fragmentation mode is a mode supporting the multi-step operation to perform batch data processing; according to the technical scheme, under the condition that the data processing operation corresponding to the to-be-processed batch task is the single-step operation, the to-be-processed batch task is executed in a simple mode, the simple mode is a mode supporting the single-step operation to carry out batch data processing, and the problem that in the prior art, service scene requirements of different complexities cannot be met at the same time is solved.

Description

Technical Field

[0001] This application relates to the field of batch task processing technology, and more specifically, to a batch task processing method, apparatus, computer-readable storage medium, and computer program product. Background Technology

[0002] Existing batch processing technologies suffer from the following key shortcomings, limiting their applicability and flexibility in complex business scenarios:

[0003] 1) Single processing mode:

[0004] Existing solutions typically only support fixed batch processing modes (such as single-threaded sequential execution or simple parallel sharding), and cannot achieve effective compatibility between high-concurrency sharding processing and flexible development modes.

[0005] The lack of adaptability to different business logics makes it difficult for the system architecture to scale in scenarios that require a mix of sharded parallel processing and custom logic execution.

[0006] 2) The sharding strategy is rigid and lacks scalability:

[0007] Traditional solutions typically rely on built-in sharding algorithms (such as fixed-size sharding or hash sharding), which cannot meet the sharding requirements under dynamic business rules or non-uniform data distribution.

[0008] The lack of support for custom sharding strategies makes it difficult to adapt in the following scenarios:

[0009] Optimized sharding in the case of data skew;

[0010] Dynamic sharding based on business rules (such as region, time, user tags);

[0011] Hybrid sharding across data sources (such as multiple databases and file systems). Summary of the Invention

[0012] The main objective of this application is to provide a batch task processing method, apparatus, computer-readable storage medium, and computer program product to at least solve the problem that the prior art cannot simultaneously meet the needs of business scenarios with different complexities.

[0013] To achieve the above objectives, according to one aspect of this application, a batch task processing method is provided, comprising: acquiring batch tasks to be processed; when the data processing operation corresponding to the batch task to be processed is a multi-step operation, executing the batch task to be processed in a sharding mode, wherein the multi-step operation includes data input, data processing, and data output, and the sharding mode is a mode that supports batch data processing of the multi-step operation; and when the data processing operation corresponding to the batch task to be processed is a single-step operation, executing the batch task to be processed in a simple mode, wherein the simple mode supports batch data processing of the single-step operation.

[0014] Optionally, the batch tasks to be processed are executed in either a sharding mode or a simple mode, including: obtaining a custom sharding strategy; generating a custom sharder based on the custom sharding strategy; and sharding the data corresponding to the batch tasks to be processed using the custom sharder and then performing the corresponding multi-step operation or single-step operation to batch process the batch tasks to be processed.

[0015] Optionally, the custom sharding is used to shard the data corresponding to the batch task to be processed, and then the corresponding multi-step operation is performed to complete the batch task to be processed. This includes: obtaining the name of the input custom sharding, wherein there are multiple custom sharding strategies, and the custom sharding corresponds one-to-one with the custom sharding strategy; loading the corresponding custom sharding according to the name of the custom sharding to switch the custom sharding strategy; and using the loaded custom sharding to shard the data corresponding to the batch task to be processed, and then performing the corresponding multi-step operation or the single-step operation to complete the batch task to be processed.

[0016] Optionally, obtaining a custom sharding strategy includes: dynamically adjusting the shard size and parallelism based on real-time data traffic and processing load to obtain the custom sharding strategy.

[0017] Optionally, the process of performing the multi-step operation includes: performing each operation of the multi-step operation using the corresponding interface.

[0018] Optionally, each operation of the multi-step operation is performed using a corresponding interface, including: reading raw data from the persistent storage system using a first interface; converting or calculating the input data using a second interface to obtain a processing result; and persisting the processing result to the target storage system using a third interface.

[0019] Optionally, the process of performing the multi-step operation includes: reading raw data from the persistent storage system; transforming or calculating the input data to obtain the processing result; and persisting the processing result to the target storage system.

[0020] Optionally, if the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode, including: if the data processing operation corresponding to the batch task to be processed is a stored procedure call, script execution, or atomic SQL operation, the batch task to be processed is executed in a simple mode.

[0021] Optionally, the batch tasks to be processed are executed in a simple mode, including: splitting the data corresponding to the batch tasks to be processed according to the fixed splitting strategy and then performing the corresponding single-step operation to process the batch tasks to be processed in batches.

[0022] According to another aspect of this application, a batch task processing apparatus is provided, comprising: an acquisition unit for acquiring batch tasks to be processed; a first execution unit for executing the batch tasks to be processed in a segmented mode when the data processing operation corresponding to the batch tasks to be processed is a multi-step operation, wherein the multi-step operation includes data input, data processing, and data output, and the segmented mode is a mode that supports batch data processing of the multi-step operation; and a second execution unit for executing the batch tasks to be processed in a simple mode when the data processing operation corresponding to the batch tasks to be processed is a single-step operation, wherein the simple mode is a mode that supports batch data processing of the single-step operation.

[0023] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0024] According to another aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements any of the methods described.

[0025] By applying the technical solution of this application, the above-mentioned batch task processing method selects different modes to process batch tasks according to different data processing operations, supports a complete solution from simple atomic operations to complex batch processing, is applicable to different business scenarios, and solves the problem that the existing technology cannot simultaneously meet the needs of business scenarios with different complexities. Attached Figure Description

[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a mobile terminal for performing a batch task processing method according to an embodiment of this application is shown.

[0028] Figure 2 A flowchart illustrating a batch task processing method according to an embodiment of this application is shown.

[0029] Figure 3 A flowchart illustrating a fragmentation mode according to an embodiment of this application is shown;

[0030] Figure 4 A structural block diagram of a batch task processing apparatus provided according to an embodiment of this application is shown.

[0031] The above figures include the following reference numerals:

[0032] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] As described in the background section, existing technologies cannot simultaneously meet the needs of business scenarios with varying degrees of complexity. To address this issue, embodiments of this application provide a batch task processing method, apparatus, computer-readable storage medium, and computer program product.

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0038] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a batch task processing method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0039] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the batch task processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] This embodiment provides a batch task processing method that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] Figure 2 This is a flowchart of a batch task processing method according to an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps:

[0042] Step S201: Obtain the batch tasks to be processed;

[0043] Specifically, developers publish batch tasks during job development and receive batch tasks to be processed.

[0044] Step S202: When the data processing operation corresponding to the batch task to be processed is a multi-step operation, the batch task to be processed is executed in a sharding mode. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports the multi-step operation for batch data processing.

[0045] Specifically, the sharding mode is suitable for complex data processing pipelines that require ensuring data consistency. It supports transactional batch processing and is suitable for complex data processing scenarios that require high throughput and data consistency guarantees. If the data processing operations corresponding to the batch tasks to be processed are multi-step operations, then the sharding mode is selected to execute the batch tasks to be processed.

[0046] Step S203: When the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode. The simple mode is a mode that supports batch data processing in the single-step operation.

[0047] Specifically, the simple mode provides lightweight atomic operation support, which is suitable for single operation scenarios such as stored procedure calls and script execution. It features low latency and high concurrency. If the data processing operation corresponding to the batch task to be processed is a single-step operation, then the simple mode is selected to execute the batch task to be processed.

[0048] In this embodiment, the batch task processing method described above selects different modes to process batch tasks according to different data processing operations, supporting a complete solution from simple atomic operations to complex batch processing, applicable to different business scenarios, and solving the problem that the existing technology cannot simultaneously meet the needs of business scenarios with different complexities.

[0049] To implement a custom sharding strategy, in one optional implementation, step S202 or step S203 above includes:

[0050] Step S2021: Obtain the custom sharding strategy;

[0051] Step S2022: Generate a custom sharder based on the above custom sharding strategy;

[0052] Step S2023: After the data corresponding to the batch tasks to be processed is split into pieces by the custom splitter, the corresponding multi-step operation or single-step operation is performed to process the batch tasks to be processed in batches.

[0053] In the above implementation, the RemotePartitioner interface is implemented: this interface defines the core methods of the sharding logic. Developers need to implement the partition() method to return a sharding strategy that meets business requirements, i.e., a custom sharding strategy. It is then registered with the Spring IOC container: the custom sharder needs to be managed as a Spring Bean to ensure it can be automatically discovered and loaded by the batch processing framework, allowing the selection of the custom sharder corresponding to the custom sharding strategy for batch processing. Standardized interfaces and configurable sharding strategies reduce code coupling and facilitate system expansion and maintenance.

[0054] In an optional implementation, to switch custom sharding strategies, step S2023 above includes:

[0055] Step S20231: Obtain the name of the custom sharder input above. There are multiple custom sharding strategies, and each custom sharder corresponds to a custom sharding strategy.

[0056] Step S20232: Load the corresponding custom shard according to the name of the custom shard to switch the custom sharding strategy.

[0057] Step S20233: After the data corresponding to the batch task to be processed is split by the loaded custom splitter, the corresponding multi-step operation or single-step operation is performed to complete the batch task to be processed.

[0058] In the above implementation, when the job starts, the bean name of the custom sharder to be used can be specified through configuration parameters (such as partition.custom.partitioner-bean-name), thereby enabling dynamic switching of custom sharding strategies and supporting switching of sharding algorithms without restarting the service.

[0059] To allow for flexible adjustments to the fragmentation strategy, in one optional implementation, step S2021 includes:

[0060] Step S20211: Dynamically adjust the shard size and parallelism based on real-time data traffic and processing load to obtain the above-mentioned custom sharding strategy.

[0061] In the above implementation, the intelligent sharding parameter configuration provides fine-grained control options, including chunk size, partition key selection, and parallelism adjustment, to adapt to different data distribution characteristics and generate custom sharding strategies that meet business needs. This allows developers to implement sharding strategies that conform to specific business rules, such as dynamic data sharding, load balancing optimization, and data skew handling.

[0062] For batch processing, in an optional implementation, step S2023 above further includes:

[0063] Step S20234: Perform each of the above multi-step operations using the corresponding interface.

[0064] In the above embodiments, each operation of the multi-step operation is processed through different interfaces, and a multi-layer processing pipeline design is adopted to realize the batch processing of complex data processing tasks.

[0065] To enable multi-step operations, in one optional implementation, step S20234 includes:

[0066] Step S202341: Read the original data from the persistent storage system using the first interface;

[0067] Step S202342: Use the second interface to transform or calculate the input data to obtain the processing result;

[0068] Step S202343: Persist the processing result to the target storage system using the third interface.

[0069] In the above implementation, the sharding mode adopts a three-layer processing pipeline design, specifically including: a data input stage, a data processing stage, and a data output stage. The data input stage (ItemStreamReader interface, i.e., the first interface) is responsible for reading raw data from a persistent storage system (including but not limited to relational databases, file systems, etc.). The data processing stage (ItemProcessor interface, i.e., the second interface) implements business logic processing units, transforming, calculating, or applying business rules to the input data. The data output stage (ItemStreamWriter interface, i.e., the third interface) persists the processing results to the target storage system. The system introduces a configurable chunkSize parameter, which defines the size of the data processing unit within the transaction boundary. Figure 3 As shown, the execution engine operates according to the following process: data is divided into blocks according to the number of records specified by `chunkSize`. Each data block completes its full read-process-write lifecycle within an independent transaction context. After the transaction is committed, the next data block is automatically processed. The sharding mode is particularly suitable for processing scenarios that meet the following characteristics: clearly separated data input, transformation, and output processes; the need to guarantee the atomicity of data processing and transaction consistency; and the requirement to support batch processing of large-scale data. Intelligent sharding and parallel processing mechanisms significantly improve the throughput of batch processing tasks, making them suitable for massive data processing scenarios.

[0070] To achieve batch processing for lightweight atomic operations, in one optional implementation, step S203 includes:

[0071] Step S2031: When the data processing operations corresponding to the batch tasks to be processed are stored procedure calls, script execution, and atomic SQL operations, the batch tasks to be processed are executed in simple mode.

[0072] In the above implementation, the simple mode features include: minimizing execution units and encapsulating single operation instructions, including stored procedure calls, script execution, and atomic SQL operations, while also supporting execution flow control via return values. It is suitable for the following situations: operations cannot be decoupled into multi-stage processing, atomic single-step operations are required, processing logic is simple, and complex data transformation is not required. The system provides a loop execution mechanism, controlling the flow in the following ways: the operation return value determines whether to continue iteration or terminate, and it supports configuring safety parameters such as the maximum number of attempts.

[0073] This application also provides a batch task processing apparatus. It should be noted that the batch task processing apparatus of this application can be used to execute the batch task processing method provided in this application. This apparatus is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0074] The following describes the batch task processing apparatus provided in the embodiments of this application.

[0075] Figure 4 This is a schematic diagram of a batch task processing apparatus according to an embodiment of this application. Figure 4 As shown, the device includes:

[0076] Acquisition unit 10 is used to acquire batch tasks to be processed;

[0077] Specifically, developers publish batch tasks during job development and receive batch tasks to be processed.

[0078] The first execution unit 20 is used to execute the batch task to be processed in a sharding mode when the data processing operation corresponding to the batch task to be processed is a multi-step operation. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports the multi-step operation for batch data processing.

[0079] Specifically, the sharding mode is suitable for complex data processing pipelines that require ensuring data consistency. It supports transactional batch processing and is suitable for complex data processing scenarios that require high throughput and data consistency guarantees. If the data processing operations corresponding to the batch tasks to be processed are multi-step operations, then the sharding mode is selected to execute the batch tasks to be processed.

[0080] The second execution unit 30 is used to execute the batch tasks to be processed in a simple mode when the data processing operation corresponding to the batch tasks to be processed is a single-step operation. The simple mode is a mode that supports batch data processing for the single-step operation.

[0081] Specifically, the simple mode provides lightweight atomic operation support, which is suitable for single operation scenarios such as stored procedure calls and script execution. It features low latency and high concurrency. If the data processing operation corresponding to the batch task to be processed is a single-step operation, then the simple mode is selected to execute the batch task to be processed.

[0082] Through this embodiment, the batch task processing device described above selects different modes to process batch tasks according to different data processing operations, supporting a complete solution from simple atomic operations to complex batch processing, applicable to different business scenarios, and solving the problem that the existing technology cannot simultaneously meet the needs of business scenarios with different complexities.

[0083] To implement a custom sharding strategy, in one optional implementation, the first execution unit or the second execution unit includes:

[0084] The acquisition module is used to obtain custom sharding strategies;

[0085] The generation module is used to generate a custom sharder based on the custom sharding strategy described above.

[0086] The first execution module is used to perform the above-mentioned multi-step operation or single-step operation after splitting the data corresponding to the batch tasks to be processed by the above-mentioned custom splitter, so as to process the batch tasks to be processed in batches.

[0087] In the above implementation, the RemotePartitioner interface is implemented: this interface defines the core methods of the sharding logic. Developers need to implement the partition() method to return a sharding strategy that meets business requirements, i.e., a custom sharding strategy. It is then registered with the Spring IOC container: the custom sharder needs to be managed as a Spring Bean to ensure it can be automatically discovered and loaded by the batch processing framework, allowing the selection of the custom sharder corresponding to the custom sharding strategy for batch processing. Standardized interfaces and configurable sharding strategies reduce code coupling and facilitate system expansion and maintenance.

[0088] In one optional implementation, to switch custom sharding strategies, the first execution module includes:

[0089] The acquisition submodule is used to acquire the name of the above-mentioned custom sharder. There are multiple custom sharding strategies, and each custom sharder corresponds to a custom sharding strategy.

[0090] The loading submodule is used to load the corresponding custom shard based on the name of the custom shard, so as to switch the custom sharding strategy.

[0091] The first execution submodule is used to perform the corresponding multi-step operation or single-step operation after splitting the data corresponding to the batch task to be processed by the loaded custom slicer.

[0092] In the above implementation, when the job starts, the bean name of the custom sharder to be used can be specified through configuration parameters (such as partition.custom.partitioner-bean-name), thereby enabling dynamic switching of custom sharding strategies and supporting switching of sharding algorithms without restarting the service.

[0093] To allow for flexible adjustments to the sharding strategy, in one optional implementation, the acquisition module includes:

[0094] The submodule is adjusted to dynamically adjust the shard size and parallelism based on real-time data traffic and processing load, resulting in the custom sharding strategy described above.

[0095] In the above implementation, the intelligent sharding parameter configuration provides fine-grained control options, including chunk size, partition key selection, and parallelism adjustment, to adapt to different data distribution characteristics and generate custom sharding strategies that meet business needs. This allows developers to implement sharding strategies that conform to specific business rules, such as dynamic data sharding, load balancing optimization, and data skew handling.

[0096] For batch processing, in one optional implementation, the first execution module further includes:

[0097] The second execution submodule is used to execute each operation of the above multi-step operation using the corresponding interface.

[0098] In the above embodiments, each operation of the multi-step operation is processed through different interfaces, and a multi-layer processing pipeline design is adopted to realize the batch processing of complex data processing tasks.

[0099] To enable multi-step operations, in one optional implementation, the second execution submodule includes:

[0100] The read subunit is used to read raw data from the persistent storage system using the first interface;

[0101] The processing subunit is used to transform or calculate the input data using the second interface to obtain the processing result;

[0102] Write sub-units are used to persist the processing results to the target storage system using a third interface.

[0103] In the above implementation, the sharding mode adopts a three-layer processing pipeline design, specifically including: a data input stage, a data processing stage, and a data output stage. The data input stage (ItemStreamReader interface, i.e., the first interface) is responsible for reading raw data from a persistent storage system (including but not limited to relational databases, file systems, etc.). The data processing stage (ItemProcessor interface, i.e., the second interface) implements business logic processing units, transforming, calculating, or applying business rules to the input data. The data output stage (ItemStreamWriter interface, i.e., the third interface) persists the processing results to the target storage system. The system introduces a configurable chunkSize parameter, which defines the size of the data processing unit within the transaction boundary. Figure 3 As shown, the execution engine operates according to the following process: data is divided into blocks according to the number of records specified by `chunkSize`. Each data block completes its full read-process-write lifecycle within an independent transaction context. After the transaction is committed, the next data block is automatically processed. The sharding mode is particularly suitable for processing scenarios that meet the following characteristics: clearly separated data input, transformation, and output processes; the need to guarantee the atomicity of data processing and transaction consistency; and the requirement to support batch processing of large-scale data. Intelligent sharding and parallel processing mechanisms significantly improve the throughput of batch processing tasks, making them suitable for massive data processing scenarios.

[0104] To enable batch processing for lightweight atomic operations, in one optional implementation, the second execution unit includes:

[0105] The second execution module is used to execute the batch tasks to be processed in a simple mode when the data processing operations corresponding to the batch tasks to be processed are stored procedure calls, script executions, and atomic SQL operations.

[0106] In the above implementation, the simple mode features include: minimizing execution units and encapsulating single operation instructions, including stored procedure calls, script execution, and atomic SQL operations, while also supporting execution flow control via return values. It is suitable for the following situations: operations cannot be decoupled into multi-stage processing, atomic single-step operations are required, processing logic is simple, and complex data transformation is not required. The system provides a loop execution mechanism, controlling the flow in the following ways: the operation return value determines whether to continue iteration or terminate, and it supports configuring safety parameters such as the maximum number of attempts.

[0107] The aforementioned batch task processing device includes a processor and a memory. The aforementioned acquisition unit, first execution unit, and second execution unit are all stored as program units in the memory, and the processor executes the aforementioned program units stored in the memory to achieve the corresponding functions. All of the aforementioned modules are located in the same processor; or, the aforementioned modules are located in different processors in any combination.

[0108] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can solve problems where existing technologies cannot simultaneously meet the needs of business scenarios with varying degrees of complexity.

[0109] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0110] This invention provides a computer-readable storage medium that includes a stored program, wherein the program, when running, controls the device containing the computer-readable storage medium to execute the batch task processing method.

[0111] Specifically, batch task processing methods include:

[0112] Step S201: Obtain the batch tasks to be processed;

[0113] Step S202: When the data processing operation corresponding to the batch task to be processed is a multi-step operation, the batch task to be processed is executed in a sharding mode. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports the multi-step operation for batch data processing.

[0114] Step S203: When the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode. The simple mode is a mode that supports batch data processing in the single-step operation.

[0115] This invention provides a processor for running a program, wherein the program executes the batch task processing method described above.

[0116] Specifically, batch task processing methods include:

[0117] Step S201: Obtain the batch tasks to be processed;

[0118] Step S202: When the data processing operation corresponding to the batch task to be processed is a multi-step operation, the batch task to be processed is executed in a sharding mode. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports the multi-step operation for batch data processing.

[0119] Step S203: When the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode. The simple mode is a mode that supports batch data processing in the single-step operation.

[0120] This invention provides a business system, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0121] Step S201: Obtain the batch tasks to be processed;

[0122] Step S202: When the data processing operation corresponding to the batch task to be processed is a multi-step operation, the batch task to be processed is executed in a sharding mode. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports the multi-step operation for batch data processing.

[0123] Step S203: When the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode. The simple mode is a mode that supports batch data processing in the single-step operation.

[0124] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0125] Step S201: Obtain the batch tasks to be processed;

[0126] Step S202: When the data processing operation corresponding to the batch task to be processed is a multi-step operation, the batch task to be processed is executed in a sharding mode. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports the multi-step operation for batch data processing.

[0127] Step S203: When the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode. The simple mode is a mode that supports batch data processing in the single-step operation.

[0128] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0138] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0139] 1) In the batch task processing method of this application, different modes are selected according to different data processing operations to process the batch tasks to be processed in batches. It supports a complete solution from simple atomic operations to complex batch processing, which is suitable for different business scenarios and solves the problem that the existing technology cannot meet the needs of business scenarios with different complexities at the same time.

[0140] 2) In the batch task processing device of this application, different modes are selected according to different data processing operations to process the batch tasks to be processed in batches. It supports a complete solution from simple atomic operations to complex batch processing, which is suitable for different business scenarios and solves the problem that the existing technology cannot meet the needs of business scenarios with different complexities at the same time.

[0141] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A batch task processing method, characterized in that, include: Get the batch of tasks to be processed; When the data processing operation corresponding to the batch task to be processed is a multi-step operation, the batch task to be processed is executed in a sharding mode. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports batch data processing of the multi-step operation. When the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode, which is a mode that supports batch data processing for single-step operations.

2. The method according to claim 1, characterized in that, Executing the batch tasks to be processed using either a sharded mode or a simple mode includes: Get a custom sharding strategy; Generate a custom sharder based on the custom sharding strategy; The custom slicer is used to slice the data corresponding to the batch tasks to be processed, and then the corresponding multi-step operation or single-step operation is performed to process the batch tasks to be processed in batches.

3. The method according to claim 2, characterized in that, After the data corresponding to the batch task to be processed is fragmented using the custom fragmenter, the corresponding multi-step operation is performed to complete the batch task to be processed, including: Obtain the name of the input custom sharder. There are multiple custom sharding strategies, and each custom sharder corresponds to a custom sharding strategy. Load the corresponding custom shard according to the name of the custom shard to switch the custom sharding strategy; The data corresponding to the batch task to be processed is split into pieces by the loaded custom splitter, and then the corresponding multi-step operation is performed to complete the batch task to be processed.

4. The method according to claim 2, characterized in that, Get custom sharding strategies, including: The custom sharding strategy is obtained by dynamically adjusting the shard size and parallelism based on real-time data traffic and processing load.

5. The method according to claim 2, characterized in that, The process of performing the multi-step operation includes: Each operation of the multi-step operation is executed using the corresponding interface.

6. The method according to claim 5, characterized in that, The operations of the multi-step operation are performed using the corresponding interface, including: The first interface is used to read the raw data from the persistent storage system; The second interface is used to transform or calculate the input data to obtain the processing result; The processing results are persisted to the target storage system using a third interface.

7. The method according to claim 1, characterized in that, When the data processing operation corresponding to the batch task to be processed is a single-step operation, the batch task to be processed is executed in a simple mode, including: When the data processing operations corresponding to the batch tasks to be processed are stored procedure calls, script execution, and atomic SQL operations, the batch tasks to be processed are executed in simple mode.

8. A batch task processing device, characterized in that, include: The acquisition unit is used to acquire batch tasks to be processed. The first execution unit is configured to execute the batch task to be processed in a sharding mode when the data processing operation corresponding to the batch task to be processed is a multi-step operation. The multi-step operation includes data input, data processing and data output. The sharding mode is a mode that supports batch data processing of the multi-step operation. The second execution unit is used to execute the batch tasks to be processed in a simple mode when the data processing operation corresponding to the batch tasks to be processed is a single-step operation. The simple mode is a mode that supports batch data processing for the single-step operation.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images