Batch service data processing method and device

Abstract

The invention provides a batch business data processing method and device, and the method comprises the steps: obtaining task description information of a to-be-scheduled task of at least one batch transaction type carried by a batch transaction request in response to the received batch transaction request sent by a client; determining at least one target micro-service according to the batch transaction type and the task description information of any to-be-scheduled task; calling at least one target micro-service to determine data information of to-be-processed batch business data in any to-be-scheduled task according to task description information of any to-be-scheduled task, and performing paging according to the data information to obtain at least one piece of paging data; and calling at least one target micro-service to load and asynchronously process the batch business data based on the at least one paging data. Therefore, the accuracy and the flexibility of task scheduling are improved, the memory pressure caused by loading excessive data at a time is avoided, and meanwhile, an asynchronous processing mechanism is adopted, so that the batch processing efficiency and the throughput capacity are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and in particular, to a batch business data processing method and device. BACKGROUND

[0002] Under the background of rapid development of the current digital economy, financial institutions, large enterprises and Internet platforms are facing growing demand for mass data processing. Various batch businesses such as salary payment, expense deduction, bill generation and report statistics are characterized by large data volume, concentrated processing period and diverse business types. With the continuous expansion of business scale and the continuous improvement of system complexity, traditional monolithic architecture has been difficult to meet the efficient, stable and scalable processing requirements. Under this trend, distributed architecture gradually becomes the mainstream technical path to support large-scale batch business processing, providing a solid technical foundation for efficient decomposition, parallel execution and unified control of tasks, and promoting the development of batch processing systems towards intelligence and platformization. SUMMARY

[0003] The present disclosure provides a batch business data processing method and device to at least partially solve one of the technical problems in the related art. The technical solution of the present disclosure is as follows: According to a first aspect of an embodiment of the present disclosure, a batch business data processing method is provided. In response to receiving a batch transaction request sent by a client, task description information of at least one to-be-scheduled task of a batch transaction type carried by the batch transaction request is obtained. According to the batch transaction type and the task description information of any to-be-scheduled task, at least one target microservice for processing the any to-be-scheduled task is determined. The at least one target microservice is called to determine data information of batch business data to be processed in the any to-be-scheduled task according to the task description information of the any to-be-scheduled task, and to perform paging according to the data information to obtain at least one paging data. The at least one paging data includes a data identifier of the batch business data. The at least one target microservice is called to load and asynchronously process the batch business data based on the at least one paging data.

[0004] According to a second aspect of the embodiments of the present disclosure, a batch service data processing apparatus is provided. A first obtaining module is configured to, in response to receiving a batch transaction request sent by a client, obtain task description information of a to-be-scheduled task of at least one batch transaction type carried by the batch transaction request. A first determining module is configured to determine at least one target microservice for processing any to-be-scheduled task according to the batch transaction type and the task description information of the to-be-scheduled task. A first processing module is configured to call the at least one target microservice to determine data information of batch service data to be processed in the any to-be-scheduled task according to the any task description information, and perform paging according to the data information to obtain at least one page data. The page data includes data identifiers of associated to-be-processed service data. A second processing module is configured to call the at least one target microservice to load and asynchronously process the batch service data based on the at least one page data.

[0005] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory storing instructions executable by the processor. The processor is configured to execute the instructions to implement the batch service data processing method according to the first aspect of the embodiments of the present disclosure.

[0006] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the batch service data processing method according to the first aspect of the embodiments of the present disclosure.

[0007] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the batch service data processing method according to the first aspect of the embodiments of the present disclosure is implemented.

[0008] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects: In this technical solution, by automatically parsing the task description information and batch transaction type from the obtained batch transaction requests, the target microservice for processing the scheduled tasks is determined, improving the accuracy of task scheduling and system flexibility. By paginating large batches of business data, memory pressure caused by loading too much data at once is avoided, ensuring stable system operation. An asynchronous parallel processing mechanism is adopted, supporting multiple paginated data to be processed simultaneously by different microservices, significantly improving processing efficiency and throughput. Without manual intervention, automated, high-concurrency, and low-latency processing of batch business data is achieved, effectively enhancing system scalability and resource utilization. Specifically, when loading and asynchronously processing batch business data, the available queue capacity of the target microservice's current worker thread is dynamically sensed, and appropriate paginated data is intelligently selected for submission, realizing efficient use of system resources. Reasonable allocation avoids thread blocking or memory overflow caused by task backlog; by submitting the adapted paginated data to an asynchronous worker thread pool, the concurrent processing capability of the asynchronous thread pool is used to asynchronously load and execute batch business sub-data, effectively improving the system's throughput and response efficiency; in addition, when loading and asynchronously processing batch business sub-data, the asynchronous worker thread pool dynamically obtains the target service instance according to the transaction type of the batch business sub-data, realizing unified access and flexible expansion of processing logic, improving system maintainability and service reusability; by automatically loading business data through the target service instance and performing message assembly and online processing one by one, fine-grained and automated execution of batch tasks in high-concurrency scenarios is achieved; the system coupling is effectively reduced, and while ensuring transaction accuracy, the processing efficiency and system stability of batch business are significantly improved.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0011] Figure 1 This is a flowchart illustrating the batch business data processing method shown in the first embodiment of this disclosure; Figure 2 This is a flowchart illustrating the batch business data processing method shown in the second embodiment of this disclosure; Figure 3 This is a flowchart illustrating the batch business data processing method shown in the third embodiment of this disclosure; Figure 4 This is a flowchart illustrating the batch business data processing method shown in the fourth embodiment of this disclosure; Figure 5 This is a schematic diagram illustrating the principle of the batch business data processing method shown in the embodiments of this disclosure; Figure 6 This is a schematic diagram illustrating the execution principle of the task to be scheduled, as shown in the embodiments of this disclosure; Figure 7 This is a schematic diagram illustrating the principle of the batch execution component shown in the embodiments of this disclosure; Figure 8 This is a schematic diagram illustrating the principle of the batch scheduling platform shown in the embodiments of this disclosure; Figure 9 This is a schematic diagram of the structure of the batch business data processing device shown in the fifth embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0013] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data are interchangeable where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0014] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0015] It's important to understand that, against the backdrop of the rapid development of the digital economy, financial institutions, large enterprises, and internet platforms are facing an ever-increasing demand for massive data processing. Various batch processing tasks, such as payroll disbursement, expense deductions, bill generation, and report statistics, are characterized by large data volumes, concentrated processing cycles, and diverse business types. As business scale continues to expand and system complexity increases, traditional monolithic architectures are no longer sufficient to meet the requirements of efficient, stable, and scalable processing. Under this trend, distributed architectures, with their flexible resource scheduling capabilities, good scalability, and high fault tolerance, are gradually becoming the mainstream technical path to support large-scale batch processing. Specifically, in a distributed microservice architecture, simply converting batch processing into batch online transaction processing is equivalent to a single online transaction interface handling hundreds of millions of data points, which may exacerbate challenges in system resource management and microservice response time, leading to system performance failing to meet business requirements.

[0016] To address at least one of the aforementioned problems, this disclosure proposes a method and apparatus for batch business data processing.

[0017] The batch business data processing method and apparatus of this disclosure are described below with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating the batch business data processing method shown in the first embodiment of this disclosure.

[0019] like Figure 1 As shown, this batch business data processing method includes the following steps: Step 101: In response to receiving a batch transaction request from the client, obtain the task description information of at least one batch transaction type of the scheduled task carried in the batch transaction request.

[0020] In this embodiment of the disclosure, when a batch transaction request is received from a client, the request content is parsed, and relevant information of one or more pending tasks is extracted. These tasks may correspond to different batch transaction types. The batch transaction types may include transaction types with multiple execution cycles under the same business scenario. For example, in the interest settlement scenario, it includes, but is not limited to, daily interest settlement, monthly interest settlement, quarterly interest settlement, and annual interest settlement. The task description information is a set of metadata describing the task to be scheduled. For example, it may include the task type, scope, rules, and key parameters required for execution, which are used to identify and schedule the task.

[0021] Step 102: Based on the batch transaction type and task description information of any scheduled task, determine at least one target microservice for processing any scheduled task.

[0022] As one possible implementation, by parsing the batch transaction type and task description information of the task to be scheduled, key routing identifiers such as business category or data characteristics are extracted. Based on preset mapping rules or service routing tables, one or more target microservices responsible for handling the task to be scheduled are automatically matched and determined, realizing dynamic binding between tasks and services, and ensuring that different types of tasks can be accurately distributed to the corresponding service instances for processing.

[0023] For example, taking any scheduled task as opening payroll accounts for employees in batches for an enterprise, based on the batch transaction type and task description information of the batch account opening request (e.g., account opening rules, account opening scope, etc.), at least one target microservice, such as the account opening microservice, is determined to handle the batch account opening request.

[0024] As an example, such as Figure 2 As shown, step 102 may include: Step 1021: Based on the batch transaction type and business parameters in the task description information of any task to be scheduled, query the pre-configured microservice transaction mapping table to determine at least one candidate microservice; wherein, the microservice transaction mapping table includes the mapping relationship between transaction types and microservices.

[0025] To decouple task scheduling from specific service implementation, one possible approach is to obtain the batch transaction type of the task (such as "daily interest settlement", "batch payroll", "monthly reconciliation", etc.) and the business parameters carried in the task description information (such as business scenario identifier, customer type, account attributes, transaction cycle, product code, etc.). This information is then used as query conditions to query a pre-configured microservice transaction mapping table. The microservice transaction mapping table is a type of structured configuration data (which can be stored in a database, configuration center, or cache). The microservice transaction mapping table includes predefined mapping entries, each of which records the address, service name, or service category of the microservice corresponding to a specific transaction type and business parameter combination. By matching the combination of transaction type and business parameters, one or more candidate microservices with processing capabilities can be accurately selected.

[0026] Step 1022: Based on the load status of each candidate microservice, determine at least one target microservice from at least one candidate microservice to process any scheduled task.

[0027] In order to achieve dynamic load balancing of scheduled tasks in a distributed environment and ensure efficient utilization of system resources and stability of scheduled task processing, in this embodiment of the disclosure, load query requests can be periodically sent to candidate microservices to obtain the load status of each candidate microservice (such as CPU utilization, request queue length, response time, etc.), and at least one target microservice for processing any scheduled task can be determined from at least one candidate microservice using a load evaluation algorithm (e.g., weighted round-robin algorithm).

[0028] Therefore, by using a microservice transaction mapping table to achieve precise task routing, it is ensured that scheduled tasks with different batch transaction types and business parameters can be matched with candidate microservices with processing capabilities, thus improving the accuracy of system scheduling. On this basis, dynamic evaluation is further performed based on the real-time load status of each candidate microservice to select the target microservice with a moderate load, effectively avoiding overload or idle resources of a single service node and improving the overall resource utilization and task processing efficiency of the system.

[0029] Step 103: Invoke at least one target microservice to determine the data information of the batch business data to be processed in any scheduled task based on the task description information of any scheduled task, and perform pagination based on the data information to obtain at least one page of data; wherein, the at least one page of data includes the data identifier of the batch business data.

[0030] In order to achieve an efficient and controllable data processing flow in batch business data processing scenarios, in this embodiment of the disclosure, the data identifier of the batch business data is divided into multiple subsets (i.e., paginated data), and each paginated data contains a unique data identifier of the batch business data (e.g., business serial number, customer ID, etc.).

[0031] To accurately paginate the identifiers of batch business data, one possible approach is to paginate the data identifiers of batch business data based on the target pagination keyword and a set page size, so as to obtain at least one page of data.

[0032] As an example, based on the target pagination keywords, batch business data is sorted to obtain a data sequence; based on the set page size, the data sequence is divided into at least one subsequence; for any subsequence, the data identifier in any subsequence is extracted, and pagination data is generated based on the data identifier in any subsequence.

[0033] In other words, the batch business data is first sorted according to the specified target pagination keywords (such as "creation time", "customer number" or "transaction amount") to ensure that the data is arranged in an orderly manner according to a certain business logic, forming a data sequence with a unified order. Then, according to the set page size (e.g., 500 items per page), the data sequence is divided into multiple continuous subsequences with a fixed length starting from the beginning. For each subsequence, the unique data identifier of each data item contained therein (such as order ID, account number, etc.) is extracted, and a pagination data is generated based on the data identifier in a subsequence.

[0034] For example, if the batch business data to be processed consists of 10,000 salary account records, the data identifiers of the batch business data are divided into 20 pages, and each page contains 500 unique data identifiers for the batch business data.

[0035] Step 104: Invoke at least one target microservice to load and asynchronously process batch business data based on at least one paginated data.

[0036] As one possible implementation, at least one microservice loads the corresponding batch business sub-data from the database or file system based on the data identifier in the received paginated data, and submits it to its own asynchronous processing thread pool for non-blocking execution.

[0037] As an example, such as Figure 3 As shown, step 104 may include: Step 1041: Using the current worker thread of any target microservice, select the first page data that matches the available queue capacity from at least one page data according to the available queue capacity of the current worker thread, and submit the first page data to the asynchronous worker thread pool of any target microservice.

[0038] To achieve dynamic adaptation between task distribution and system processing capacity, and to avoid queue overflow or resource overload due to task backlog, in this embodiment, the available capacity of the task queue associated with the current worker thread of the target microservice is obtained in real time. Based on this capacity value, a first page of data matching the size is selected from multiple pages of data to be processed, ensuring that the amount of submitted tasks does not exceed the processing capacity. Subsequently, the first page of data is submitted to the asynchronous worker thread pool of the target microservice, where idle threads in the asynchronous worker thread pool asynchronously execute subsequent data loading and business processing operations.

[0039] Step 1042: Using an asynchronous worker thread pool, load and asynchronously process batch business sub-data corresponding to the data identifiers recorded in the first page data.

[0040] To achieve standardization, scalability, and high concurrency in batch processing, one possible approach is to use an asynchronous worker thread pool to dynamically obtain target service instances that implement preset interfaces based on the batch transaction type. These instances then complete the data loading, message assembly, and online processing for each transaction.

[0041] As an example, an asynchronous worker thread pool is used to obtain a target service instance from any target microservice that is used to process batch transaction types and implements a preset batch processing interface, based on the batch transaction type to which the batch business sub-data belongs. The preset batch processing interface is used to define the loading operation of business data and the transaction processing flow. The target service instance is used to load the batch business sub-data and perform transaction message assembly and online transaction processing on each loaded batch business sub-data.

[0042] In other words, firstly, based on the batch transaction type to which the data belongs (such as "monthly interest settlement" or "quarterly interest settlement"), the target service instance that handles the batch transaction type and implements the preset batch processing interface is found in the target microservice. The preset batch processing interface standardizes the loading method, processing flow, and execution contract of business data, ensuring the standardization of various transaction processing logics. Then, based on the target service instance, the specific batch business sub-data is loaded, and each piece of data in the batch business sub-data independently performs transaction message assembly (such as generating request messages that conform to the communication protocol) and online transaction processing.

[0043] In this embodiment, by dynamically selecting appropriate paginated data for submission based on the available queue capacity of the current worker thread, queue blocking or resource waste caused by task overload is avoided, achieving a reasonable match between processing capacity and task load. After the selected paginated data is submitted to the asynchronous worker thread pool, the system can concurrently load and process the batch business sub-data associated with the corresponding data identifier in a non-blocking manner, improving the overall processing throughput. By combining pagination scheduling with asynchronous execution, memory usage is effectively controlled, preventing system performance degradation due to excessive data volume, while enhancing system stability, response speed, and resource utilization.

[0044] Furthermore, in some embodiments, such as Figure 4 As shown, after calling at least one target microservice to load and asynchronously process batch business data based on at least one paginated data, the batch business data processing method may further include: Step 105: During the batch business data processing, periodically perform statistics on the execution status of at least one target microservice loading and asynchronously processing batch business data to obtain statistical results.

[0045] To promptly understand the processing progress and system operation status of batch business data, as a possible implementation method, during the batch business data processing, the system actively collects the processing status of each target microservice on the batch business data at set time intervals (such as every 30 seconds or every minute), including the number of completed subtasks, the number of tasks being processed, the number of failed or retried tasks, and the processing speed (data per second). This data is then summarized and analyzed to form statistical results. It should be noted that these statistical results can be used to display the processing progress in real time, generate monitoring reports, or trigger alarm mechanisms.

[0046] Step 106: Send statistical results to the client. The statistical results are used for progress visualization tracking and / or anomaly alarms in the workflow topology where any scheduled task is located. The workflow topology is used to indicate the processing flow of scheduled tasks of different batch transaction types.

[0047] To enable users to intuitively grasp the overall execution status of tasks to be scheduled, one possible approach is to send statistical results to the client. These results include the real-time processing status of each task, such as the amount of data processed, success rate, number of failures, processing speed, and processing time. Based on this data, the client dynamically displays the execution progress of each task node in the workflow topology diagram, visually presenting the entire batch processing flow. When a node is detected to have processing delays, an excessive failure rate, or task blocking, the system can automatically trigger an alarm. The workflow topology describes the dependencies and execution paths between different batch transaction types of tasks. For example, the workflow topology includes process structures such as "data validation, pagination scheduling, asynchronous processing, and result aggregation."

[0048] In order to accurately obtain the workflow topology of the scheduled tasks that match different batch transaction types, in this embodiment of the disclosure, in response to receiving a batch transaction operation triggered by the target object, the batch transaction type selected by the target object when uploading the batch transaction file is obtained; and according to the batch transaction type, the workflow topology of the scheduled tasks that match the batch transaction type is loaded.

[0049] In other words, when a batch transaction operation is received from a target object (such as a user or business system), the system retrieves the batch transaction type selected by the target object when uploading the batch transaction file (e.g., "batch payroll" or "monthly interest settlement"). Subsequently, the system loads the workflow topology of the scheduled tasks matching the batch transaction type from the predefined configuration. This topology describes the complete processing flow that the scheduled tasks should go through, including the task execution order, dependencies, participating microservice nodes, and processing rules for each stage.

[0050] As an example, during batch business data processing, the execution status of loading and asynchronously processing batch business data by at least one target microservice is periodically statistically analyzed to obtain statistical results. By monitoring the execution status of batch business data in real time, backlogs or processing anomalies can be quickly identified. The statistical results are sent to the client, where they are used for progress visualization tracking and / or anomaly alerts in the workflow topology of any scheduled task. The workflow topology is used to indicate the processing flow of scheduled tasks of different batch transaction types. The client can obtain the processing progress information of scheduled tasks of different batch transaction types in the workflow topology in real time, presented in an intuitive visualization form to enhance transparency.

[0051] To achieve precise control over the processing of batch business sub-data, dynamic management and control of the processing of batch business sub-data is achieved by responding to batch pause commands, batch resume commands, batch restart commands, and batch cancel commands sent by the client.

[0052] In this embodiment of the disclosure, in response to receiving a batch control instruction sent by a client, wherein the batch control instruction includes a batch pause instruction, the processing of batch service sub-data corresponding to the data identifier recorded in the second page data associated with the batch pause instruction is paused; in response to receiving a batch control instruction sent by a client, wherein the batch control instruction includes a batch resume instruction, the processing of the paused batch service sub-data corresponding to the data identifier recorded in the third page data associated with the batch resume instruction is resumed; in response to receiving a batch control instruction sent by a client, wherein the batch control instruction includes a batch rerun instruction, the processing of batch service sub-data corresponding to the data identifier recorded in the fourth page data associated with the batch rerun instruction is reprocessed; and in response to receiving a batch control instruction sent by a client, wherein the batch control instruction includes a batch cancel instruction, the processing of batch service sub-data corresponding to the data identifier recorded in the fifth page data associated with the batch cancel instruction is canceled.

[0053] For example, the system is processing a batch of 100,000 payroll transactions. Having processed the 60,000th transaction, maintenance personnel discover a temporary failure in the downstream payment channel. To prevent subsequent transaction failures, they send a "batch pause command" through the client, specifying to pause processing of the pages corresponding to data numbers 60001 to 100000 (i.e., the second page data). The system immediately stops processing these unprocessed data sub-data, preserving the current progress and avoiding a large number of failure records. Another example: after the payroll task is paused, the payment channel failure is repaired. Maintenance personnel send a "batch resume command" through the client, linking it to the previously paused page data (the third page data, corresponding to data numbers 60001 to 100000). Upon receiving the command, the system resumes processing the remaining business sub-data in that page, continuing from transaction number 60001 until all are completed. Yet another example: a certain... The interest calculation for the first 5000 transactions was inaccurate due to an incorrect interest rate parameter configuration. The administrator sent a "batch rerun command" via the client, specifying the page to be reprocessed (the fourth page, containing data identifiers for the first 5000 accounts). The system reloaded this business sub-data, executed the correct interest calculation logic, and overwrote or corrected the original results, ensuring data accuracy. Similarly, a company uploaded a "batch refund" request containing 3000 transactions. The system had begun processing the first 500 transactions. Later, due to business adjustments, the company decided to cancel all refunds. The administrator sent a "batch cancellation command" via the client, linking it to the unprocessed page data (the fifth page, corresponding to data identifiers 501 to 3000). The system immediately terminated processing of the business sub-data corresponding to these data identifiers, marking the task as "cancelled," preventing erroneous refunds and ensuring business security.

[0054] In summary, by automatically parsing the task description information and batch transaction type from the batch transaction requests, the target microservice for processing the scheduled tasks is determined, improving the accuracy of task scheduling and the system's flexibility. Paging large batches of business data avoids memory pressure caused by loading too much data at once, ensuring stable system operation. The asynchronous parallel processing mechanism supports multiple paginated data being processed simultaneously by different microservices, significantly improving processing efficiency and throughput. Without manual intervention, automated, high-concurrency, and low-latency processing of batch transactions is achieved, effectively enhancing the system's scalability and resource utilization.

[0055] In addition, to clearly illustrate the batch business data processing method, Figure 5 This is a schematic diagram illustrating the principle of the batch business data processing method shown in the fifth embodiment of this disclosure.

[0056] In the embodiments disclosed herein, such as Figure 6As shown, taking the application of batch business data processing methods to a business system as an example, the business system may include: a transaction center, a batch scheduling platform, and a batch execution component.

[0057] In the embodiments disclosed herein, such as Figure 5 and Figure 6 As shown, operations and maintenance personnel trigger components in the operations and maintenance management platform interface used to start batch data processing, such as the "Start" button, to send batch transaction requests to the business system. It should be noted that the batch transaction request can carry task description information for at least one batch transaction type of task to be scheduled. After receiving the batch transaction request, the business system's transaction center can determine at least one target microservice (e.g., microservices 1, 2, and 3) to process the task based on its batch transaction type and task description information. When the batch transaction interface provided by the target microservice is called, pagination is performed on the batch business data to be processed and the data is persisted. In the background asynchronous worker thread of the batch execution component in the business system, this persisted paginated data is loaded, and then the online transaction for the corresponding single data is executed. The batch execution component can also be divided into two modules: a synchronously called pagination processor and an asynchronously polled pagination executor.

[0058] The synchronous pagination processor aggregates an abstract and public batch operation interface (BatchProcessorHandler), which is implemented by various microservices. This interface can be used for tasks such as paginating batch data based on pagination keywords (split), loading paginated data (load), and assembling single messages to call online transactions (process). When developing batch transactions, core business system microservices only need to implement the relevant functions of the BatchProcessorHandler interface according to their respective business characteristics. In the synchronous pagination processor, it calls the pagination function (split) implemented by the microservice to complete data pagination and persists the paginated data to the database. The asynchronous paging executor is a background worker thread. In a distributed cloud-native environment, each container has only one paging executor. Based on the available queue size for the current thread, it loads a certain amount of paging data from the database using the splitLoader and places it into an asynchronous worker thread pool for parallel processing. When processing paging data, the asynchronous thread pool obtains the BatchProcessorHandler interface implemented by the microservice based on the corresponding batch transaction flag. It then calls the load function to load the business data using the page information recorded in the paging data, and finally calls the process function to assemble the transaction message and execute a single online transaction. Furthermore, after processing a single paging data item, the post-processing operation (afterProcess) of the common template can perform checks such as whether the current paging is complete, whether all paging items in the current batch transaction are complete, and whether the current transaction failed. This allows for the acquisition of various indicator data for the batch transaction, making the batch transaction process observable.

[0059] In other words, such as Figure 7 As shown, by calling the pagination method (e.g., split()) of the pagination processor (e.g., BatchPageSplitHandler) in the batch execution component layer, the data information of the batch business data corresponding to the scheduled task is split into pages. Then, through the application extension batch processor in the application extension layer, the data information before pagination is preprocessed (e.g., cleaned, transformed, or filtered) by calling beforeSplit(), and the paginated data after pagination is postprocessed (e.g., supplemented, validated, or adjusted) by calling afterSplit(). The post-processed paginated data is persisted to the database (e.g., OceanBase) for subsequent loading. Then, through the pagination executor (e.g., BatchScheduler), the paginated data to be processed is read from the database in a round-robin fashion until all paginated data has been processed. In the pagination executor, a pagination loader (e.g., SplitLoader) loads specific paginated data from the database. Each paginated data is assigned a unique pagination sequence number. Then, each paginated data is handed over to worker threads in an asynchronous thread pool for parallel processing. Each worker thread loads the data information of the paginated data according to the pagination sequence number (e.g., using the load() method). Finally, online transaction processing (e.g., process) is called to perform single-transaction processing, that is, processing each piece of data information in the paginated data one by one. Each piece of data information follows the standard online transaction processing flow to ensure consistency. Finally, post-processing (e.g., afterProcess) is called to update the status of the current paginated data and count metrics such as the number of successes / failures.

[0060] It should be noted that, in order to address the issue of how to observe and control the tasks to be scheduled, this disclosure can realize the observation and control of the tasks to be scheduled based on a batch scheduling platform. Figure 8 This is a schematic diagram of the principle of the batch scheduling platform shown in the seventh embodiment of this disclosure. The principle of the batch scheduling platform is as follows: The operation management platform sends various management instructions to the batch scheduling platform via message queues (MQ), such as instructions to start and complete tasks. The batch scheduling platform receives these management instructions from the operation management platform and performs batch control on the current status of the tasks to be scheduled (e.g., various batch control instructions). It continuously checks the current status of the tasks to be scheduled (e.g., tasks not completed, timed out, or abnormal) through a loop, and sends the current status back to the operation management platform. This allows for early warning management and timeout management based on the current status, enabling the observation of the batch business data corresponding to the tasks to be scheduled. In this embodiment, the core application receives batch control instructions from the batch scheduling platform, implements specific business logic through application extension interfaces, performs online transactions of batch business data one by one through online transaction interfaces, a paging processor handles the paging and splitting of batch business data, and a paging executor processes each page of paging data through a loop until all pages are completed.

[0061] Corresponding to the batch business data processing method provided in the above embodiments, this disclosure also provides a batch business data processing apparatus. Since the batch business data processing apparatus provided in this disclosure corresponds to the batch business data processing method provided in the above embodiments, the implementation of the batch business data processing method is also applicable to the batch business data processing apparatus provided in this disclosure, and will not be described in detail in this disclosure.

[0062] Figure 9 This is a schematic diagram of the structure of the batch business data processing device shown in the fifth embodiment of this disclosure.

[0063] like Figure 9 As shown, the batch business data processing device 900 includes: a first acquisition module 901, a first determination module 902, a first processing module 903, and a second processing module 904.

[0064] The system comprises the following modules: a first acquisition module 901, configured to acquire task description information of at least one batch transaction type of scheduled task carried in the batch transaction request in response to receiving a batch transaction request from a client; a first determination module 902, configured to determine at least one target microservice for processing any scheduled task based on the batch transaction type and task description information of any scheduled task; a first processing module 903, configured to call at least one target microservice to determine the data information of the batch business data to be processed in any scheduled task based on any task description information, and to perform pagination based on the data information to obtain at least one paginated data; wherein the paginated data includes the data identifier of the associated business data to be processed; and a second processing module 904, configured to call at least one target microservice to load and asynchronously process the batch business data based on at least one paginated data.

[0065] As one possible implementation of this disclosure, the second processing module 904 is configured to select, based on the available queue capacity of the current worker thread of any target microservice, first page data that matches the available queue capacity from at least one page data, and submit the first page data to the asynchronous worker thread pool of any target microservice; and load and asynchronously process batch business sub-data corresponding to the data identifier recorded in the first page data through the asynchronous worker thread pool based on the first page data.

[0066] As one possible implementation of this disclosure, the second processing module 904 is used to obtain a target service instance in any target microservice that is used to process batch transaction types and implements a preset batch processing interface, based on the batch transaction type to which the batch business sub-data belongs, through an asynchronous worker thread pool; wherein, the preset batch processing interface is used to define the loading operation of business data and the transaction processing flow; using the target service instance, the batch business sub-data is loaded, and transaction message assembly and online transaction processing are performed on each loaded batch business sub-data.

[0067] As one possible implementation of this disclosure, the first processing module 903 is used to obtain target pagination keywords associated with the business to which the batch business data belongs; and to paginate the data identifiers of the batch business data using the target pagination keywords and a set page size to obtain at least one paginated data.

[0068] As one possible implementation of this disclosure, the first processing module 903 is used to sort batch business data based on target pagination keywords to obtain a data sequence; divide the data sequence into at least one subsequence based on a set page size; extract data identifiers from any subsequence, and generate pagination data based on the data identifiers from any subsequence.

[0069] As one possible implementation of this disclosure, the first determining module 902 is configured to query a pre-configured microservice transaction mapping table based on the batch transaction type and business parameters in the task description information of any task to be scheduled, and determine at least one candidate microservice; wherein, the microservice transaction mapping table includes the mapping relationship between transaction types and microservices; and determine at least one target microservice from the at least one candidate microservice for processing any task to be scheduled based on the load status of each candidate microservice.

[0070] As one possible implementation of this disclosure, the batch business data processing device 900 further includes a statistics module and a sending module.

[0071] The statistics module is used to periodically collect statistics on the execution status of at least one target microservice loading and asynchronously processing batch business data during batch business data processing, so as to obtain statistical results; the sending module is used to send the statistical results to the client, wherein the statistical results are used for progress visualization tracking and / or anomaly alarms in the workflow topology where any scheduled task is located; the workflow topology is used to indicate the processing flow of scheduled tasks of different batch transaction types.

[0072] As one possible implementation of this disclosure, the batch business data processing device 900 further includes: a second acquisition module and a loading module.

[0073] The second acquisition module is used to acquire the batch transaction type selected by the target object when uploading the batch transaction file in response to receiving a batch transaction operation triggered by the target object; the loading module is used to load the workflow topology of the scheduled task that matches the batch transaction type according to the batch transaction type.

[0074] As one possible implementation of this disclosure, the batch business data processing device 900 further includes: a third processing module. The third processing module is configured to perform at least one of the following: In response to receiving a batch control instruction sent by the client, wherein the batch control instruction includes a batch pause instruction, suspending the processing of batch service sub-data corresponding to the data identifier recorded in the second page data associated with the batch pause instruction; In response to receiving a batch control instruction sent by the client, wherein the batch control instruction includes a batch resume instruction, resuming the processing of the suspended batch service sub-data corresponding to the data identifier recorded in the third page data associated with the batch resume instruction; In response to receiving a batch control instruction sent by the client, wherein the batch control instruction includes a batch rerun instruction, reprocessing the batch service sub-data corresponding to the data identifier recorded in the fourth page data associated with the batch rerun instruction; In response to receiving a batch control instruction sent by the client, wherein the batch control instruction includes a batch cancellation instruction, canceling the processing of the batch service sub-data corresponding to the data identifier recorded in the fifth page data associated with the batch cancellation instruction.

[0075] The batch business data processing apparatus of this disclosure improves the accuracy of task scheduling and system flexibility by intelligently routing tasks to the corresponding target microservices based on the task description information and batch transaction type in the automatically parsed batch transaction requests. By paging large batches of business data, it avoids memory pressure caused by loading too much data at once, ensuring stable system operation. The asynchronous parallel processing mechanism supports multiple paginated data being processed simultaneously by different microservices, significantly improving processing efficiency and throughput. Without manual intervention, it achieves automated, high-concurrency, and low-latency processing of batch business data, effectively enhancing system scalability and resource utilization.

[0076] In an exemplary embodiment, an electronic device is also proposed.

[0077] The electronic devices include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the batch business data processing method proposed in any of the foregoing embodiments.

[0078] As an example, Figure 10 This is a schematic diagram of the structure of an electronic device 1000 as shown in an exemplary embodiment of this disclosure, as follows: Figure 10 As shown, the aforementioned electronic device 1000 may further include: The memory 1010 and the processor 1020 are connected by a bus 1030, which connects different components (including the memory 1010 and the processor 1020). The memory 1010 stores a computer program, and when the processor 1020 executes the program, it implements the batch business data processing method described in this embodiment of the disclosure.

[0079] Bus 1030 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0080] Electronic device 1000 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 1000, including volatile and non-volatile media, removable and non-removable media.

[0081] The memory 1010 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1040 and / or cache memory 1050. The server may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 1060 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 1030 via one or more data media interfaces. Memory 1010 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0082] A program / utility 1080 having a set (at least one) of program modules 1070 may be stored in, for example, memory 1010. Such program modules 1070 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1070 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0083] Electronic device 1000 can also communicate with one or more external devices 1090 (e.g., keyboard, pointing device, display 1091, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 1092. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1093. As shown, network adapter 1093 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0084] The processor 1020 performs various functional applications and data processing by running programs stored in the memory 1010.

[0085] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the batch business data processing method of this disclosure embodiment, and will not be repeated here.

[0086] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to perform the batch business data processing method proposed in any of the above embodiments. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0087] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, which, when executed by a processor, implement the batch business data processing method proposed in any of the above embodiments.

[0088] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0089] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for batch business data processing, characterized in that, include: In response to receiving a batch transaction request from a client, the task description information of at least one batch transaction type of the scheduled task carried in the batch transaction request is obtained; Based on the batch transaction type and task description information of any scheduled task, determine at least one target microservice for processing the scheduled task. The at least one target microservice is invoked to determine the data information of the batch business data to be processed in any scheduled task based on the task description information of any scheduled task, and to perform pagination based on the data information to obtain at least one paginated data; wherein, the at least one paginated data includes the data identifier of the batch business data; The at least one target microservice is invoked to load and asynchronously process the batch business data based on the at least one paginated data.

2. The method according to claim 1, characterized in that, The invocation of the at least one target microservice to load and asynchronously process the batch business data based on the at least one paginated data includes: Based on the available queue capacity of the current worker thread of any target microservice, select the first page data that matches the available queue capacity from the at least one page data, and submit the first page data to the asynchronous worker thread pool of any target microservice. The asynchronous worker thread pool loads and asynchronously processes batch business sub-data corresponding to the data identifiers recorded in the first pagination data, based on the first pagination data.

3. The method according to claim 2, characterized in that, The step of loading and asynchronously processing batch business sub-data corresponding to the data identifier recorded in the first pagination data through the asynchronous worker thread pool includes: The asynchronous worker thread pool obtains a target service instance from any target microservice that processes the batch transaction type and implements a preset batch processing interface, based on the batch transaction type to which the batch business sub-data belongs; wherein, the preset batch processing interface is used to define the loading operation of business data and the transaction processing flow. Using the target service instance, the batch business sub-data is loaded, and transaction message assembly and online transaction processing are performed on each loaded batch business sub-data.

4. The method according to claim 1, characterized in that, The at least one paginated data is generated using the following steps: Obtain the target pagination keywords associated with the business to which the batch of business data belongs; Using the target pagination keyword and the set page size, the data identifiers of the batch business data are paginated to obtain at least one page of data.

5. The method according to claim 4, characterized in that, The step of using the target pagination keyword and a set page size to paginate the data identifiers of the batch business data to obtain at least one paginated data includes: Based on the target pagination keywords, the batch business data is sorted to obtain a data sequence; Based on the set page size, the data sequence is divided into at least one subsequence; For any subsequence, extract the data identifier from the subsequence, and generate the pagination data based on the data identifier from the subsequence.

6. The method according to claim 1, characterized in that, The step of determining at least one target microservice for processing any scheduled task based on the batch transaction type and task description information of any scheduled task includes: Based on the batch transaction type and business parameters in the task description information of any scheduled task, a pre-configured microservice transaction mapping table is queried to determine at least one candidate microservice; wherein, the microservice transaction mapping table includes the mapping relationship between transaction types and microservices; Based on the load status of each candidate microservice, at least one target microservice is determined from the at least one candidate microservice to process any scheduled task.

7. The method according to claim 1, characterized in that, The method further includes: During the batch business data processing process, the execution status of the at least one target microservice loading and asynchronously processing the batch business data is periodically statistically analyzed to obtain statistical results. The statistical results are sent to the client, wherein the statistical results are used for progress visualization tracking and / or anomaly alarms in the workflow topology where any of the scheduled tasks are located; the workflow topology is used to indicate the processing flow of scheduled tasks of different batch transaction types.

8. The method according to claim 7, characterized in that, The workflow topology is obtained using the following steps: In response to receiving a batch transaction operation triggered by the target object, the batch transaction type selected by the target object when uploading the batch transaction file is obtained; Based on the batch transaction type, load the workflow topology of the scheduled tasks that match the batch transaction type.

9. The method according to claim 1, characterized in that, The method further includes at least one of the following: In response to receiving a batch control instruction sent by the client, wherein the batch control instruction includes a batch pause instruction, the processing of batch business sub-data corresponding to the data identifier recorded in the second page data associated with the batch pause instruction is paused; In response to receiving a batch control instruction sent by the client, and the batch control instruction includes a batch resume instruction, the batch service sub-data corresponding to the data identifier recorded in the third page data associated with the batch resume instruction that has been paused is resumed. In response to receiving a batch control instruction sent by the client, and the batch control instruction includes a batch rerun instruction, the batch service sub-data corresponding to the data identifier recorded in the fourth page data associated with the batch rerun instruction is reprocessed. In response to receiving a batch control instruction sent by the client, and the batch control instruction includes a batch cancellation instruction, the batch service sub-data corresponding to the data identifier recorded in the fifth page data associated with the batch cancellation instruction is cancelled.

10. A batch business data processing device, characterized in that, include: The first acquisition module is used to, in response to receiving a batch transaction request sent by the client, acquire task description information of at least one batch transaction type of scheduled task carried in the batch transaction request; The first determining module is used to determine at least one target microservice for processing any scheduled task based on the batch transaction type of any scheduled task and the task description information. The first processing module is used to call the at least one target microservice to determine the data information of the batch business data to be processed in any scheduled task according to the task description information, and to perform pagination according to the data information to obtain at least one paginated data; wherein, the paginated data includes the data identifier of the associated business data to be processed; The second processing module is used to call the at least one target microservice to load and asynchronously process the batch business data based on the at least one paginated data.

Images