A parallel optimization method and system for batch data processing

Abstract

The present disclosure relates to a parallel optimization method and system for batch data processing, the method comprising: receiving batch input heterogeneous input data; preprocessing the received heterogeneous input data to obtain preprocessed heterogeneous input data; dynamically dividing the preprocessed heterogeneous input data into multiple task shards, and then generating an ordered task shard execution queue based on all the task shards; wherein each task shard is independently executed and labeled with a corresponding unique shard identifier; reading the task shard execution queue, real-time detecting the current available computing resources, distributing each task shard to an idle resource node in the current available computing resources, and starting multi-thread parallel execution; executing the task shard distributed thereon, and monitoring the execution status of each task shard; aggregating the execution results of all the task shards, merging them into a complete result set based on the order of the original task shards, exporting the complete result set into a pre-set specified format, and generating a task execution report based on the complete result set.

Description

Technical Field

[0001] This disclosure relates to the fields of big data processing and artificial intelligence technology, and more specifically, to a parallel optimization method and system for batch data processing. Background Technology

[0002] With the deepening of digital transformation, the volume of data required for processing in scenarios such as daily enterprise operations, AI model training, and data report generation is growing exponentially, with single batches often exceeding one million records. This places extremely high demands on data processing speed, resource consumption, and stability. At the same time, batch processing tasks such as large model inference, text cleaning, and format conversion are becoming increasingly common. Traditional serial processing and fixed-slice processing technologies cannot dynamically allocate computing resources such as CPUs and GPUs according to the data volume and task complexity, and can no longer meet the needs of efficient processing. They generally suffer from slow processing speed, high costs, and susceptibility to interruptions.

[0003] Current technologies for batch data processing mainly suffer from the following technical problems: (1) Low task scheduling efficiency and unbalanced resource allocation: Existing batch data processing mostly adopts serial execution or fixed sharding mode, which cannot dynamically allocate computing resources such as CPU and GPU according to the data volume and task complexity. When multiple tasks are concurrent, resource contention and waste of idle resources are very likely to occur, resulting in slow overall processing speed and inability to adapt to the needs of processing massive amounts of data of millions or more.

[0004] (2) Excessive resource consumption and high repetitive overhead in model inference: Existing large-scale model batch inference mostly adopts a single input and single call mode. The model needs to be loaded and unloaded repeatedly, resulting in high GPU / CPU memory consumption, high inference latency, high hardware cost, and no support for mixed scheduling of local and cloud models, resulting in extremely low resource utilization.

[0005] (3) Lack of task fault tolerance mechanism and poor overall stability: Traditional batch processing solutions lack the ability to capture fine-grained task exceptions. If a single subtask fails to execute, data is abnormal, or the interface times out, it will directly cause the entire batch of tasks to be interrupted and rolled back. It is impossible to handle abnormal tasks individually. All tasks need to be re-executed, which is time-consuming, labor-intensive, and prone to data loss.

[0006] (4) Weak data adaptation and process scalability: The existing solution only supports fixed format data input, has poor adaptability to multiple data formats such as Excel, CSV, and JSONL, and has no standardized task sharding rules, making it difficult to customize and adjust. It cannot adapt to batch processing needs in different scenarios, and the cost of later function expansion and process optimization is high.

[0007] In summary, existing batch data processing solutions have significant technical shortcomings in terms of task scheduling efficiency, model inference resource consumption, task fault tolerance mechanisms, data adaptation, and process scalability, and urgently require technical solutions to address these shortcomings. Summary of the Invention

[0008] To address the problems existing in the prior art, this disclosure proposes a parallel optimization method and system for batch data processing to solve at least one of the aforementioned technical problems. The technical solution adopted in this disclosure is as follows: In a first aspect, this disclosure provides a parallel optimization method for batch data processing, comprising the following steps: Step S100: Receive heterogeneous input data in batches; Step S200: Preprocess the received heterogeneous input data to obtain preprocessed heterogeneous input data; Step S300: First, the preprocessed heterogeneous input data is dynamically divided into multiple task fragments, and then an ordered task fragment execution queue is generated based on all task fragments; wherein, each task fragment is executed independently and marked with a corresponding unique fragment identifier; Step S400: Read the task shard execution queue, detect the currently available computing resources in real time, and allocate each task shard to the idle resource nodes in the currently available computing resources according to the principle of "high-priority tasks first and load balancing", and start multi-threaded parallel execution; Step S500: Execute the task shards allocated to each resource node and monitor the execution status of each task shard; Step S600: After all normally executed task fragments have completed their execution, summarize the execution results of all task fragments, merge them into a complete result set based on the original order of each task fragment, export the complete result set in a preset specified format, and generate a task execution report based on the complete result set.

[0009] Optionally, heterogeneous input data can include data in multiple formats such as Excel, CSV, and JSONL.

[0010] Optionally, in step S200, the preprocessing includes duplicate data filtering, format error data filtering, null value cleanup, field format standardization, and encoding rule unification, thereby ensuring that subsequent data processing is compliant and avoiding invalid tasks from consuming resources.

[0011] Optionally, in step S300, the preprocessed heterogeneous input data is first divided into multiple task fragments based on the total amount of data, the complexity of a single data item, and the total amount of computing resources.

[0012] Each task fragment is executed independently as an independent execution unit, and is marked with a unique fragment identifier to facilitate subsequent scheduling and anomaly tracking.

[0013] Optionally, in step S400, the resource usage during the multi-threaded parallel execution process is monitored in real time. If an overload is detected on a resource node, the tasks to be executed on that resource node are fragmented and transferred to other idle resource nodes, thereby eliminating resource conflicts and ensuring parallel processing efficiency.

[0014] Optionally, in step S400, the computing resources include a central processing unit, a graphics processing unit, and memory hardware resources, etc. Optionally, in step S500, if a task segment contains an artificial intelligence large model inference task, all data within the task segment is integrated into a batch input data package, and a unified batch inference interface is called to complete model loading and full data inference in one go.

[0015] Furthermore, in step S500, during the inference process, the locally deployed model and the cloud application programming interface (API) are automatically adapted and flexibly switched according to task requirements, thereby balancing inference speed and cost.

[0016] Optionally, in step S500, in response to detecting a task segment that is executing abnormally, a retry is initiated for the task segment. If the retry is successful, execution continues. If the retry fails, the task segment is marked as an abnormal segment, the reason for the abnormality and the corresponding data are recorded, and the task segment is skipped to continue executing other task segments without interrupting the execution of the overall batch in which the task segment is located.

[0017] Optionally, abnormal execution includes at least one of execution failure, data error, and interface timeout.

[0018] By monitoring and capturing abnormal execution such as task execution errors, data errors, and interface timeouts, retrying and logging exceptions, and skipping task fragments with abnormal execution, it ensures that individual abnormal fragments will not affect the execution of other task fragments, and that the global execution will not be affected by local problems. Subsequent individual abnormal fragments can be rerun individually without having to re-execute all tasks because of individual abnormal fragments, further improving the flexibility and reliability of batch data processing.

[0019] Optionally, the specified format includes Excel or JSONL format.

[0020] Optionally, the task execution report includes information such as total data volume, number of successfully processed items, number of abnormal items, total execution time, and details of abnormal fragments, which facilitates user verification and separate rerun of abnormal tasks.

[0021] A second aspect of this disclosure provides a parallel optimization system for batch data processing, the system comprising: The data input module is used to receive heterogeneous input data in batches; The data preprocessing module is used to preprocess the received heterogeneous input data to obtain preprocessed heterogeneous input data; The dynamic sharding module is used to first dynamically shard the preprocessed heterogeneous input data into multiple task shards, and then generate an ordered task shard execution queue based on all task shards; each task shard is executed independently and marked with a corresponding unique shard identifier. The resource scheduling module is used to read the task shard execution queue, detect the currently available computing resources in real time, and allocate each task shard to the idle resource nodes in the currently available computing resources according to the principle of "high-priority tasks first and load balancing", and start multi-threaded parallel execution; The execution monitoring module is used to execute the task shards allocated to each resource node and monitor the execution status of each task shard. The execution results module is used to summarize the execution results of all task segments after all normally executed task segments have completed their execution. It merges the results into a complete result set based on the original order of each task segment, exports the complete result set to a preset specified format, and generates a task execution report based on the complete result set.

[0022] In a third aspect, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the parallel optimization method for batch data processing as described above.

[0023] In a fourth aspect, this disclosure provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the parallel optimization method for batch data processing as described above.

[0024] The beneficial effects of this disclosure are as follows: This disclosure provides a parallel optimization method and system for batch data processing. Through standardized sharding, dynamic resource allocation, a unified inference interface, and an exception sharding retry skip mechanism, it achieves efficient, low-consumption, and stable processing of massive data. It forms a working mechanism of "dynamic sharding - resource scheduling - batch inference - limited retries for exceptions - exception sharding skip," creating a batch data processing parallel optimization system with dynamic resource scheduling, low-consumption batch inference, high fault tolerance, and full format adaptation. It can adapt to various batch data processing scenarios such as data cleaning, batch report generation, text classification, and model training data preprocessing, solving the core pain points of existing solutions such as low efficiency, high resource consumption, and poor stability. This achieves faster, lower-cost, and stable operation of massive data batch processing. Unlike traditional fixed sharding and fixed resource allocation technologies, this disclosure uses dynamic sharding and resource scheduling to achieve automated dynamic sharding based on data volume and task complexity. It integrates hardware and software resources to form a resource pool, dynamically allocates and schedules resources, and achieves load balancing and parallel execution of multiple tasks, improving processing efficiency and resource utilization.

[0025] This disclosure can interface with standardized batch inference interfaces, optimizing single-item, multi-stage inference into unified batch inference across multiple data sets. This reduces the overhead of repeated model loading and invocation, supports hybrid scheduling of local and cloud models, and lowers hardware resource consumption. This disclosure implements independent exception capture, limited-number automatic retries, and exception segment marking for individual task segments. Failure of individual exception segments does not interrupt the overall batch task; it supports individual reruns of exception segments, solving the pain point of batch interruption and rollback due to partial failures in traditional solutions. This disclosure is compatible with various commonly used batch data formats, configured to automatically complete data cleaning and standardization processing, automatically merge segment results and generate a complete execution report after task completion, achieving fully automated processing without manual intervention, and realizing intelligent and precise management of batch data processing.

[0026] Compared with the prior art, this disclosure has the following advantages: (1) Significantly improved processing efficiency: Through dynamic task sharding and parallel scheduling, the overall data processing efficiency has been improved by more than 80%, which can support the sharding and synchronous execution of millions of data in seconds, greatly shortening the batch processing time.

[0027] (2) Significantly reduced resource consumption: Through batch unified inference optimization, the overhead of repeated model loading is reduced, GPU / CPU resource consumption is reduced by 45%, hardware costs and energy consumption decrease simultaneously, and resource utilization is maximized.

[0028] (3) Overall improvement in task stability: Through task-level exception tolerance and retry mechanism, the failure of a single task does not affect the overall batch operation, the task execution stability is improved by 95%, and full rollback and repeated operations are avoided.

[0029] (4) Enhanced adaptability and scalability: By being compatible with multiple input data formats, it supports custom sharding rules and hybrid model scheduling, enabling rapid adaptation to various batch data processing scenarios. Subsequent expansion does not require refactoring of the underlying logic.

[0030] (5) More convenient operation and maintenance and troubleshooting: Complete execution report and anomaly tracking functions can quickly locate abnormal data and the cause of the problem. It supports rerunning of abnormal segments separately without having to reprocess all data, which greatly reduces the cost of operation and maintenance and the difficulty of troubleshooting. Attached Figure Description

[0031] The accompanying drawings, which form part of this application, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0032] Figure 1 This is a flowchart of the parallel optimization method for batch data processing described in Embodiment 1 of this disclosure.

[0033] Figure 2 This is an architecture diagram of the parallel optimization system for batch data processing described in Embodiment 2 of this disclosure. Detailed Implementation

[0034] The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0035] The following detailed descriptions are exemplary and intended to provide further detailed explanation of this disclosure. Unless otherwise specified, all technical terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure.

[0036] Currently, existing batch data processing solutions mainly fall into two categories: The first type is the traditional serial batch processing scheme. This scheme treats the entire batch of data as a single processing unit, executing it line by line and batch by batch in the order of data entry. It lacks a task partitioning mechanism, allocates fixed computing resources, occupies a single hardware resource throughout the entire process, and outputs the results uniformly after all data processing is complete. It has no ability to handle exceptions in segments. This type of scheme suffers from problems such as extremely slow processing speed, exponential growth in processing time for large batches of data, inability to dynamically allocate resources, inability to reuse idle hardware, lack of fault tolerance mechanism, and the direct failure of the entire batch due to a single data exception. It cannot adapt to the needs of large-scale data processing.

[0037] The second category is the fixed-sharding parallel processing scheme. This scheme manually divides the data into shards of a fixed number of records and distributes them to multiple basic threads for parallel execution. Some schemes support basic model calls, but the sharding rules are fixed, resource allocation is rigid, and there is no dynamic adjustment capability. Anomaly handling only supports full batch retries. This type of scheme suffers from problems such as rigid sharding rules, inability to adapt to tasks of different complexities, inflexible resource scheduling, congestion in high-complexity tasks, resource waste in low-complexity tasks, high resource consumption due to repeated calls in model inference, high cost of anomaly retries, and no separate mechanism for rerunning abnormal tasks. Overall, it lacks flexibility and stability.

[0038] Example 1: like Figure 1 As shown, this disclosure provides a parallel optimization method for batch data processing, the method comprising steps S100 to S600.

[0039] Step S100: Receive heterogeneous input data in batches.

[0040] Furthermore, heterogeneous input data can include data in multiple formats such as Excel, CSV, and JSONL.

[0041] Step S200: Preprocess the received heterogeneous input data to obtain preprocessed heterogeneous input data.

[0042] Furthermore, in step S200, the preprocessing includes duplicate data filtering, format error data filtering, null value cleanup, field format standardization, and encoding rule unification, thereby ensuring that subsequent data processing is compliant and avoiding invalid tasks from consuming resources.

[0043] Step S300: First, the preprocessed heterogeneous input data is dynamically divided into multiple task fragments, and then an ordered task fragment execution queue is generated based on all task fragments; wherein, each task fragment is executed independently and marked with a corresponding unique fragment identifier.

[0044] Furthermore, in step S300, based on the total amount of data, the complexity of a single data item, and the total amount of computing resources, the preprocessed heterogeneous input data is first divided into multiple task fragments.

[0045] Furthermore, in step S300, the dynamic sharding follows the rule that each task shard has 1000 data entries by default, and is also configured to support customizing the number of data entries in each task shard.

[0046] Each task fragment is executed independently as an independent execution unit, and is marked with a unique fragment identifier to facilitate subsequent scheduling and anomaly tracking.

[0047] Step S400: Read the task shard execution queue, detect the currently available computing resources in real time, and allocate each task shard to the idle resource nodes in the currently available computing resources according to the principle of "high-priority tasks first and load balancing", and start multi-threaded parallel execution.

[0048] Furthermore, in step S400, the resource usage during the multi-threaded parallel execution process is monitored in real time; If an overload is detected on a resource node, the tasks to be executed on that resource node are fragmented and transferred to other idle resource nodes, thereby eliminating resource conflicts and ensuring parallel processing efficiency.

[0049] Furthermore, in step S400, the computing resources include a central processing unit, a graphics processing unit, and memory hardware resources, etc. Furthermore, by connecting local and cloud computing resource pools, the aforementioned hardware resources can be integrated to form a unified resource pool.

[0050] Step S500: Execute the task shards allocated to each resource node and monitor the execution status of each task shard.

[0051] Furthermore, in step S500, if a task segment contains an artificial intelligence large model inference task, all data within the task segment is integrated into a batch input data packet, and a unified batch inference interface is called to complete model loading and full data inference in one go.

[0052] Furthermore, in step S500, during the inference process, the locally deployed model and the cloud application programming interface (API) are automatically adapted and flexibly switched according to task requirements, thereby balancing inference speed and cost.

[0053] Furthermore, in step S500, in response to detecting a task segment that is executing abnormally, a retry is initiated for the task segment. If the retry is successful, execution continues. If the retry fails, the task segment is marked as an abnormal segment, the reason for the abnormality and the corresponding data are recorded, and the task segment is skipped to continue executing other task segments without interrupting the execution of the overall batch in which the task segment is located. Abnormal execution includes at least one of the following: execution failure, data error, and interface timeout.

[0054] By monitoring and capturing abnormal execution such as task execution errors, data errors, and interface timeouts, retrying and logging exceptions, and skipping task fragments with abnormal execution, it ensures that individual abnormal fragments will not affect the execution of other task fragments, and that the global execution will not be affected by local problems. Subsequent individual abnormal fragments can be rerun individually without having to re-execute all tasks because of individual abnormal fragments, further improving the flexibility and reliability of batch data processing.

[0055] Furthermore, in step S500, a task shard experiencing abnormal execution is retried up to 3 times, and the task shard that fails 3 consecutive retries is marked as an abnormal shard. It should be noted that the consecutive 3 retry failures here refer to a scenario where, after abnormal execution, 3 consecutive retries fail due to the same abnormal reason. If a task shard succeeds in 3 or fewer retries due to a certain abnormal reason, and then retries are initiated again due to other abnormal reasons, the retries for these two abnormal reasons are not accumulated; only the retries caused by a single abnormal reason are accumulated. For example, if a task shard successfully initiates 3 retries due to interface timeout (the first 2 retries fail), and then fails again due to data abnormality during execution, this does not constitute 3 consecutive retry failures due to the same abnormal reason.

[0056] Furthermore, in step S500, the task sharding queue can be connected to various large artificial intelligence model interfaces, the batch inference input and output format can be uniformly defined, the batch inference call interface can be encapsulated, and multiple sets of data can be loaded uniformly and inferred at once, thereby reducing the overhead of repeated model loading.

[0057] Step S600: After all normally executed task fragments have completed their execution, summarize the execution results of all task fragments, merge them into a complete result set based on the original order of each task fragment, export the complete result set in a preset specified format, and generate a task execution report based on the complete result set.

[0058] Furthermore, the specified format may include Excel or JSONL format.

[0059] Furthermore, the task execution report may include information such as total data volume, number of successfully processed items, number of abnormal items, total execution time, and details of abnormal fragments, which facilitates user verification and separate rerun of abnormal tasks.

[0060] The working principle of this disclosure can be understood through the following example of batch AI sentiment analysis of millions of customer reviews from an enterprise. The complete execution process is as follows: Step (100) Data Import: Receive a CSV file containing 1 million customer reviews uploaded in batches by operators.

[0061] Step (200) Data preprocessing: The received CSV format file is preprocessed, including filtering duplicate comments and cleaning up null values, to obtain 980,000 valid data entries after preprocessing.

[0062] Step (300) Dynamic sharding: According to the rule of 1,000 data points per task shard, the 980,000 valid data points are divided into 980 task shards. An ordered execution queue is generated based on the 980 task shards, and each task shard is marked with a unique shard identifier.

[0063] Step (400) Resource scheduling: Read the task slice execution queue, detect 4 groups of available GPU resources in real time, and allocate each task slice to the idle resource node in the currently available computing resources according to the principle of "high priority task first, load balancing". Distribute the 980 task slices evenly to the 4 groups of available GPU resources and start multi-threaded parallel execution.

[0064] Step (500) Execution monitoring: Execute the task slices allocated to them using available GPU resources, and monitor the execution status of each task slice.

[0065] In response to the detection of 15 task shards with abnormal execution, up to 3 retries are initiated for each of the abnormal task shards. The task shards that succeed in the retry continue to be executed. Among them, 2 task shards are marked as abnormal shards because of individual comments with abnormal format. These 2 abnormal shards are skipped and the remaining shards are processed.

[0066] In this process, the data from each task segment is integrated into a batch input data package, which calls the unified sentiment analysis big data model batch interface to complete the sentiment determination of 1,000 comments in a single segment at once. The sentiment analysis big data model only needs to be loaded once to complete the analysis and determination of all comments in the same task segment, thereby significantly reducing resource consumption.

[0067] Step (600) Output of execution results: Summarize the execution results of each task segment, export the complete sentiment analysis result table (i.e., the complete result set), and generate a task execution report.

[0068] The task execution report can simultaneously mark two failed task segments, allowing operators to rerun the failed segments individually later. Throughout the implementation, no task needed to be re-executed due to failed task segments, avoiding wasted computing resources and reducing overall time by 80% compared to traditional methods.

[0069] Example 2: like Figure 2 As shown, in a second aspect, this disclosure provides a parallel optimization system for batch data processing, the system comprising: The data input module 100 is used to receive heterogeneous input data in batches; The data preprocessing module 200 is used to preprocess the received heterogeneous input data to obtain preprocessed heterogeneous input data; The dynamic sharding module 300 is used to first dynamically shard the preprocessed heterogeneous input data into multiple task shards, and then generate an ordered task shard execution queue based on all task shards; wherein, each task shard is executed independently and marked with a corresponding unique shard identifier. The resource scheduling module 400 is used to read the task slice execution queue, detect the currently available computing resources in real time, and allocate each task slice to the idle resource nodes in the currently available computing resources according to the principle of "high priority tasks first and load balancing", and start multi-threaded parallel execution; The execution monitoring module 500 is used to execute the task shards allocated to each resource node and monitor the execution status of each task shard. The execution result module 600 is used to summarize the execution results of all task segments after all normally executed task segments have completed their execution, merge them into a complete result set based on the original order of each task segment, export the complete result set to a preset specified format, and generate a task execution report based on the complete result set.

[0070] The data input module 100, target detection module 200, relative position module 300, penetration depth module 400, anomaly determination module 500 and historical data module 600 in Example 2 can correspond to S100, S200, S300, S400, S500 and S600 in Example 1, respectively.

[0071] It should be noted that the system described in Embodiment 2 is only one system implementation of the method described in Embodiment 1, and does not mean that the method described in Embodiment 1 necessarily depends on the system described in Embodiment 2.

[0072] The system described in Example 2 can be implemented using a cloud / local hybrid resource scheduling + visual backend operation scheme, or it can be implemented using the following three alternative schemes. As long as the logic for accepting batch heterogeneous input data, preprocessing, dynamic sharding, batch inference, and exception tolerance remains unchanged, only the execution carrier or operation mode is adjusted, the technical objectives of the entire disclosure can be achieved. Specifically, it can be as follows: (1) Pure local server deployment alternative: The entire parallel optimization system is deployed on the enterprise's local server cluster, without relying on cloud resources and external network environment. All data processing, sharding scheduling, and inference execution are completed locally, and data is stored on the local server. It is suitable for enterprise scenarios with extremely high data privacy and security requirements and prohibition of external network transmission. The core processing flow is completely consistent with the original solution.

[0073] (2) Command line script execution alternative: For technical operators, a command line execution version is provided, which completes data import, sharding parameter configuration, task start and result export through script commands, replacing the visual background operation, adapting to automated script integration and batch timed processing scenarios, and the underlying core optimization logic is no different from the original solution.

[0074] (3). Containerized microservice alternative: Each functional module is encapsulated as an independent Docker microservice container. Data preprocessing, resource scheduling, inference optimization, and exception fault tolerance modules are deployed independently and scaled up as needed. Module collaboration is achieved through container orchestration tools, which can adapt to high-concurrency and large-scale batch processing scenarios of large enterprises, while keeping the module functions and core processes unchanged.

[0075] Example 3: Embodiment 3 of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the parallel optimization method for batch data processing as described in Embodiment 1.

[0076] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0077] Example 4: Embodiment 4 of this disclosure provides an electronic device including a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the parallel optimization method for batch data processing described in Embodiment 1.

[0078] Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0079] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0084] In summary, the parallel optimization method and system for batch data processing provided in embodiments 1-4 of this disclosure achieve efficient, low-consumption, and stable processing of massive data through standardized sharding, dynamic resource allocation, unified inference interface, and an abnormal sharding retry skip mechanism. Based on the working mechanism of "dynamic sharding - resource scheduling - batch inference - limited retry for abnormalities - abnormal sharding skip," this disclosure forms a batch data processing parallel optimization system with dynamic resource scheduling, low-consumption batch inference, high fault tolerance, and full format adaptation. It can adapt to various batch data processing scenarios such as data cleaning, batch report generation, text classification, and model training data preprocessing, solving the core pain points of existing solutions such as low efficiency, high resource consumption, and poor stability, and achieving accelerated cost reduction and stable operation of massive data batch processing. Unlike traditional fixed sharding and fixed resource allocation technologies, this disclosure achieves automated dynamic sharding based on data volume and task complexity through dynamic sharding and resource scheduling. It integrates hardware and software resources to form a resource pool, dynamically allocates and schedules resources, and achieves multi-task load balancing and parallel execution, improving processing efficiency and resource utilization.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit them. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this disclosure. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this disclosure should be covered within the protection scope of the claims of this disclosure.

Claims

1. A parallel optimization method for batch data processing, characterized in that, Includes the following steps: Step S100: Receive heterogeneous input data in batches; Step S200: Preprocess the received heterogeneous input data to obtain preprocessed heterogeneous input data; Step S300: First, the preprocessed heterogeneous input data is dynamically divided into multiple task fragments, and then an ordered task fragment execution queue is generated based on all task fragments; wherein, each task fragment is executed independently and marked with a corresponding unique fragment identifier; Step S400: Read the task shard execution queue, detect the currently available computing resources in real time, and allocate each task shard to the idle resource nodes in the currently available computing resources according to the principle of "high priority tasks first, load balancing", and start multi-threaded parallel execution; Step S500: Execute the task shards allocated to each resource node and monitor the execution status of each task shard; Step S600: After all normally executed task fragments have completed their execution, summarize the execution results of all task fragments, merge them into a complete result set based on the original order of each task fragment, export the complete result set in a preset specified format, and generate a task execution report based on the complete result set.

2. The parallel optimization method for batch data processing according to claim 1, characterized in that, In step S200, the preprocessing includes duplicate data filtering, format error data filtering, null value cleanup, field format standardization, and encoding rule unification.

3. The parallel optimization method for batch data processing according to claim 1, characterized in that, In step S300, based on the total amount of data, the complexity of a single data item, and the total amount of computing resources, the preprocessed heterogeneous input data is first divided into multiple task fragments.

4. The parallel optimization method for batch data processing according to claim 1, characterized in that, In step S400 Real-time monitoring of resource usage during multi-threaded parallel execution; If an overload is detected on a resource node, the tasks to be executed on that resource node will be fragmented and transferred to other idle resource nodes.

5. The parallel optimization method for batch data processing according to claim 1, characterized in that, In step S400, the computing resources include a central processing unit, a graphics processing unit, and memory hardware resources.

6. The parallel optimization method for batch data processing according to claim 1, characterized in that, In step S500, if a task segment contains an artificial intelligence large model inference task, all data in the task segment is integrated into a batch input data package, and a unified batch inference interface is called to complete the model loading and full data inference in one go.

7. The parallel optimization method for batch data processing according to claim 1, characterized in that, In step S500, in response to detecting a task segment that is executing abnormally, a retry is initiated for the task segment. If the retry is successful, execution continues. If the retry fails, the task segment is marked as an abnormal segment, the reason for the abnormality and the corresponding data are recorded, and the task segment is skipped to continue executing other task segments without interrupting the execution of the overall batch in which the task segment is located. Abnormal execution includes at least one of execution failure, data error, and interface timeout.

8. The parallel optimization method for batch data processing according to claim 7, characterized in that, In step S500, a maximum of 3 retries are initiated for the task fragment that has an abnormal execution, and the task fragment that fails to retries 3 times in a row is marked as an abnormal fragment.

9. The parallel optimization method for batch data processing according to claim 1, characterized in that, The specified format includes Excel or JSONL format; The task execution report includes the total data volume, the number of successfully processed records, the number of abnormal records, the total execution time, and details of abnormal fragments.

10. A parallel optimization system for batch data processing, characterized in that, The system also includes: The data input module (100) is used to receive heterogeneous input data in batches; The data preprocessing module (200) is used to preprocess the received heterogeneous input data to obtain preprocessed heterogeneous input data; The dynamic sharding module (300) is used to first dynamically shard the preprocessed heterogeneous input data into multiple task shards, and then generate an ordered task shard execution queue based on all task shards; wherein, each task shard is executed independently and marked with a corresponding unique shard identifier; The resource scheduling module (400) is used to read the task slice execution queue, detect the available computing resources in real time, and allocate each task slice to the idle resource nodes in the available computing resources according to the principle of "high priority tasks first, load balancing", and start multi-threaded parallel execution; The execution monitoring module (500) is used to execute the task shards allocated to each resource node and monitor the execution status of each task shard. The execution result module (600) is used to summarize the execution results of all task segments after all normally executed task segments have completed execution, merge them into a complete result set based on the original order of each task segment, export the complete result set in a preset specified format, and generate a task execution report based on the complete result set.

Images