Large file batch processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a large file batch processing method and device, computer equipment, a computer readable storage medium and a computer program product. Comprising the following steps: controlling a public unit to read a to-be-split source file from a file storage server, and extracting file features of the source file; based on the file features, controlling the public unit to perform first segmentation, second segmentation and third segmentation on the to-be-split source file to obtain a plurality of unit small files; based on a preset routing mapping relationship, controlling the public unit to distribute the plurality of unit small files to the corresponding service units; controlling the service unit to process the plurality of unit small files to obtain a plurality of result files, and transmitting the plurality of result files to a file storage server; and controlling the public unit to read the plurality of result files from the file storage server, and merging the plurality of result files to obtain a service result file. By adopting the method provided by the invention, the batch processing efficiency and throughput of the large files can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of file processing, in particular to a batch processing method and device for large files, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Files become the main carriers of data exchange between bank core systems and peripheral systems due to their general format and convenient transmission. Core systems usually read and process business data in files uploaded by peripheral systems in batch processing (batch running) mode. To ensure the normal development of the next day's business, batch processing of full large files needs to be completed efficiently within a limited window period. However, with the rapid expansion of the scale of bank business and customer groups, business data is growing explosively, leading to a double increase in the data volume of single files and the number of unprocessed files, and the timeliness of batch running is facing severe challenges. The current batch running scheme under the traditional centralized architecture tries to improve the efficiency of batch processing of large files by means of database expansion, multi-server parallel job, or building a file processing time-consuming model and optimizing batch file task scheduling strategy.

[0003] The batch processing method of large files under the traditional centralized architecture faces the problems of long task time consumption and low throughput. SUMMARY

[0004] Therefore, it is necessary to provide a batch processing method, device, computer device, computer readable storage medium and computer program product for large files, which can improve the efficiency and throughput of batch processing of large files.

[0005] In a first aspect, the present application provides a batch processing method for large files, comprising:

[0006] controlling a public unit to read a source file to be split from a file storage server and extracting file features of the source file;

[0007] controlling the public unit to perform first, second and third splitting on the source file to be split based on the file features, to obtain a plurality of unit small files;

[0008] controlling the public unit to distribute the plurality of unit small files to corresponding business units based on a preset routing mapping relationship; controlling the business units to process the plurality of unit small files to obtain a plurality of result files, and transmitting the plurality of result files to the file storage server;

[0009] controlling the public unit to read the plurality of result files from the file storage server, and merging the plurality of result files to obtain a business result file.

[0010] In one of the embodiments, based on the file features, the common unit controls the first segmentation, the second segmentation and the third segmentation of the source file to be split to obtain a plurality of unit small files, including:

[0011] Based on the size data in the file features, the common unit controls the first segmentation of the source file to obtain a plurality of quantitative block files; based on the service data in the file features and the preset routing mapping relationship, the common unit controls the second segmentation of the plurality of quantitative block files to obtain unit large files corresponding to different service units; based on the preset evaluation index and the resource limited situation of different service units, the common unit controls the third segmentation of the unit large files to obtain a plurality of unit small files.

[0012] In one of the embodiments, based on the preset routing mapping relationship, the common unit distributes the plurality of unit small files to the corresponding service units, including:

[0013] The common unit reads the routing identification information of the plurality of unit small files; matches the routing identification information with the preset routing mapping relationship to determine the target service unit corresponding to each unit small file; and sends the plurality of unit small files to the corresponding target service unit.

[0014] In one of the embodiments, the method further includes:

[0015] Based on the preset algorithm, the service unit controls the processing of the plurality of unit small files to obtain a file unique identification; for any unit small file, based on the file unique identification, the processing state of the unit small file is judged; if the processing state is unprocessed, the service logic processing of the unit small file is executed to obtain a result file.

[0016] In one of the embodiments, the method further includes:

[0017] If the processing state is processed, the service logic processing of the unit small file is skipped.

[0018] In one of the embodiments, after extracting the file features of the source file, the method further includes:

[0019] Based on the preset algorithm, the processing of the source file is performed to obtain a unique identification of the source file; based on the unique identification, the processing state of the source file is judged; if the processing state is unprocessed, the subsequent file feature extraction is performed; if the processing state is processed, the processing of the source file is skipped, and the common unit reads the source file to be split from the file storage server.

[0020] In a second aspect, the application further provides a batch processing device of a large file, including:

[0021] read a source file to be split from the file storage server and extract file features of the source file by the public unit;

[0022] perform first splitting, second splitting and third splitting on the source file to be split based on the file features by the public unit to obtain a plurality of unit small files;

[0023] distribute the plurality of unit small files to corresponding business units based on a preset routing mapping relationship by the public unit; control the business units to process the plurality of unit small files to obtain a plurality of result files, and transmit the plurality of result files to the file storage server;

[0024] read the plurality of result files from the file storage server by the public unit, and merge the plurality of result files to obtain a business result file.

[0025] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0026] read a source file to be split from the file storage server and extract file features of the source file by the public unit;

[0027] perform first splitting, second splitting and third splitting on the source file to be split based on the file features by the public unit to obtain a plurality of unit small files;

[0028] distribute the plurality of unit small files to corresponding business units based on a preset routing mapping relationship by the public unit; control the business units to process the plurality of unit small files to obtain a plurality of result files, and transmit the plurality of result files to the file storage server;

[0029] read the plurality of result files from the file storage server by the public unit, and merge the plurality of result files to obtain a business result file.

[0030] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0031] read a source file to be split from the file storage server and extract file features of the source file by the public unit;

[0032] perform first splitting, second splitting and third splitting on the source file to be split based on the file features by the public unit to obtain a plurality of unit small files;

[0033] Based on the preset routing mapping relationship, the control public unit distributes multiple unit small files to the corresponding business units; the control business units process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server;

[0034] The control unit reads multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

[0035] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0036] The control unit reads the source file to be split from the file storage server and extracts the file characteristics of the source file;

[0037] Based on file characteristics, the control unit performs a first split, a second split, and a third split on the source file to be split, resulting in multiple smaller unit files;

[0038] Based on the preset routing mapping relationship, the control public unit distributes multiple unit small files to the corresponding business units; the control business units process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server;

[0039] The control unit reads multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

[0040] The aforementioned batch processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product for large files first involves a control unit reading the source file to be split from a file storage server and extracting its file characteristics. Then, based on these characteristics, the control unit performs a first, second, and third split on the source file to be split, resulting in multiple smaller unit files. Next, based on a preset routing mapping relationship, the control unit distributes these smaller unit files to corresponding business units. The business units process these smaller unit files to obtain multiple result files, which are then transmitted to the file storage server. Finally, the control unit reads these result files from the file storage server and merges them to obtain a final business result file. By combining a unitized architecture with a dynamic multi-layered splitting algorithm, and applying global routing and parallel processing mechanisms, the performance bottleneck of traditional centralized architectures can be overcome, significantly improving the efficiency and throughput of batch processing for large files. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a batch processing method for large files in one embodiment;

[0043] Figure 2 This is a detailed flowchart illustrating a batch processing method for large files in one embodiment;

[0044] Figure 3 This is a structural block diagram of a batch processing device for large files in one embodiment.

[0045] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0048] In one embodiment, such as Figure 1 As shown, a batch processing method for large files is provided. This embodiment illustrates the method using a terminal as an example. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0049] Step 102: Control the common unit to read the source file to be split from the file storage server and extract the file characteristics of the source file.

[0050] The common unit pre-deploys file splitting services and a common data storage module, obtaining the binary data stream or text content of the source file to be split from the file storage server through a file reading interface. The common unit stores and deploys all common data and public services involved in the batch processing of files across multiple business units. Batch processing of large files encompasses core processes such as file splitting, file downloading, file processing, file merging, and file uploading. Under the unitized architecture, common operations with low correlation to business characteristics, such as file splitting, file merging, and file uploading, are centrally deployed and maintained, avoiding redundant storage of general data and repeated deployment of general services by business units, thus reducing redundant development of general capabilities. Furthermore, due to the diversity of transaction scenarios in peripheral systems, the types of unprocessed data received by the core system also exhibit diverse characteristics. Considering that the source files in the file batch processing model may not contain characteristic data that directly indicates the business unit, a complete key field mapping table needs to be established to maintain the mapping relationship between various other key business fields and fields that can be directly used to calculate business unit parameters, thereby supporting business unit routing and database sharding processing of diverse batch file data. The file features include at least the source file's size data, business identification data (such as customer number range, business type field), data row count features, and basic attributes such as file format and encoding type. During the extraction process, a unique identifier for the source file is generated simultaneously using the MD5 hash algorithm (Message-Digest Algorithm 5) and stored in a public file temporary table for anti-duplicate verification in subsequent splitting stages. If the source file is encrypted or compressed, the public unit first performs decryption and decompression preprocessing before extracting file features.

[0051] Step 104: Based on file characteristics, control the common unit to perform the first, second, and third splits on the source file to be split, resulting in multiple unit small files.

[0052] The splitting process is achieved through a file splitting service within a common unit. Source file data uploaded from external systems is typically unordered, and a single file may contain various types of unprocessed data with different business characteristics. Under the unitized architecture of the core system, multiple types of business data need to be distributed to different business units for processing; therefore, the business characteristics of file data are defined as one of the key features of the file. The unordered nature of the file data also prevents the model from vertically splitting the file using specific rules to divide data belonging to different business units into different decomposed files. Therefore, it is necessary to match routing mapping relationships line by line based on the key business information carried by the source file's line data to determine the business unit to which each line of file data belongs. Since the reading time of file line data depends on the order of magnitude of the data contained in the file, the size of the source file is also defined as a key feature affecting the splitting. Furthermore, in addition to the unordered nature of the source file data, the amount of data with the same business characteristics is also randomly distributed, resulting in randomness in the size of each unit file derived from the source file. Considering that the smallest unit for batch processing of files is a single data entry within the file, the resulting small unit files will directly affect the concurrency of the file processing stage; therefore, the number of lines in a unit file is also defined as a key feature affecting the batch processing of large files in this model.

[0053] Step 106: Based on the preset routing mapping relationship, control the public unit to distribute multiple unit small files to the corresponding business units; control the business units to process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server.

[0054] The preset routing mapping relationship is a unique unit routing mapping with the path "data shard - data database - business unit". The common unit reads the business identification information of the unit small file through the global routing service, matches the mapping relationship to determine the target business unit, and performs distribution and scheduling in combination with the load status of the business unit cluster (CPU utilization, memory usage, current number of tasks).

[0055] Step 108: The control unit reads multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

[0056] The common unit scans all unit result files associated with the source file in the file storage server through the file merging service, and classifies and matches them based on the source file identifier (such as MD5 code). The merging process follows preset file merging rules (such as sorting by data generation time, business serial number, or splicing according to the source file splitting order) to integrate the content of the result files. After the merging is completed, a complete business result file is generated, and the file integrity and data consistency are verified synchronously (such as checking the total number of records and checksum). Finally, the business result file is stored in the specified path of the file storage server, and the processing status of the corresponding source file in the common file temporary table is updated to "completed". At the same time, a merging log is generated for audit traceability.

[0057] The aforementioned batch processing method for large files first involves the control unit reading the source file to be split from the file storage server and extracting its file characteristics. Then, based on these characteristics, the control unit performs a first, second, and third split on the source file, resulting in multiple smaller unit files. Next, based on a pre-defined routing mapping, the control unit distributes these smaller unit files to their corresponding business units. Each business unit processes these smaller unit files, generating multiple result files, which are then transmitted to the file storage server. Finally, the control unit reads these result files from the file storage server and merges them to obtain the final business result file. By combining a unitized architecture with a dynamic multi-layered splitting algorithm, and applying global routing and parallel processing mechanisms, the performance bottleneck of traditional centralized architectures can be overcome, significantly improving the efficiency and throughput of batch processing for large files.

[0058] In an exemplary embodiment, based on file characteristics, the control unit performs a first segmentation, a second segmentation, and a third segmentation on the source file to be split, resulting in multiple smaller unit files, including:

[0059] Based on the size data in the file features, the control unit performs a first segmentation on the source file to obtain multiple quantitative block files; based on the business data in the file features and the preset routing mapping relationship, the control unit performs a second segmentation on the multiple quantitative block files to obtain unit large files corresponding to different business units; based on preset evaluation indicators and the resource constraints of different business units, the control unit performs a third segmentation on the unit large files to obtain multiple unit small files.

[0060] For example, based on the size data in the file characteristics, the granularity of the segmentation is dynamically matched according to the principle of "small to small, large to large". The first segmentation splits the source file into a fixed number of block files of a preset size. Based on the business identification data in the file characteristics, combined with the preset "data sharding - data partitioning - business unit" routing mapping relationship, the fixed number of block files are read serially and the business affiliation of each line of data is identified for the second segmentation, and the data of the same business unit are classified and grouped into unit large files. Based on the data row count feature in the file characteristics and the evaluation results of the parallel processing capability of the business unit, the task time, I / O throughput and resource distribution uniformity are selected as indicators to match an appropriate number of split rows for each unit large file, and the third segmentation is performed to split it into fixed-row unit small files. During the segmentation process, an MD5 unique identifier is generated for each unit small file and recorded in a public file temporary table to achieve full-process traceability.

[0061] In this embodiment, a three-level progressive segmentation strategy is used to achieve fine-grained decomposition of large files, effectively avoiding the problems of I / O blocking and excessive memory consumption caused by loading ultra-large files at once. It accurately matches the business processing boundaries under the unitized architecture, ensures the business relevance and unit isolation of data processing, and maximizes the resource carrying capacity and parallel processing potential of each unit.

[0062] In an exemplary embodiment, based on a preset routing mapping relationship, the common unit controls the distribution of multiple unit small files to the corresponding business units, including:

[0063] The control unit reads the routing identification information of multiple unit files; matches the routing identification information with the preset routing mapping relationship to determine the target service unit corresponding to each unit file; and sends the multiple unit files to the corresponding target service unit.

[0064] For example, the routing identification information includes business characteristic fields (such as customer number range, business type code) carried in the unit file and the unit belonging mark in the file metadata; the public unit uses the matching algorithm (such as hash matching, range matching) in the global routing service to accurately match the routing identification information with the mapping table; after determining the target business unit, the public unit obtains the cluster load status (CPU utilization, memory usage, number of currently unprocessed tasks) of each business unit in real time, and uses a load balancing strategy (such as round-robin, weighted distribution) to push the unit file to the local storage node or message queue of the target business unit; during the file sending process, a distribution log is generated synchronously to record information such as file identifier, target unit, and sending time, and the distribution status in the public file temporary table is updated.

[0065] In this embodiment, the routing distribution process is based on identifier matching and load adaptation. By using clear routing identifier information and unique mapping relationships, it achieves precise targeting of small unit files to target business units. Combined with a dynamic scheduling strategy based on real-time load status, it avoids processing delays caused by task backlog in high-load units and makes full use of resource redundancy in low-load units, thereby improving overall distribution efficiency and system resource utilization.

[0066] In one exemplary embodiment, the control business unit processes multiple small unit files to obtain multiple result files, including:

[0067] Based on a preset algorithm, the control business unit processes multiple small unit files to obtain a unique file identifier. For any small unit file, the processing status of the small unit file is determined based on the unique file identifier. If the processing status is unprocessed, the business logic processing is performed on the small unit file to obtain the result file.

[0068] Optionally, the default algorithm can be either the MD5 hash algorithm or the SHA-256 algorithm (Secure Hash Algorithm 256, a 256-bit version of the secure hash algorithm). The processing status includes both unprocessed and processed.

[0069] The process involves a file storage server matching and downloading small unit files, using the MD5 algorithm to encode the file content and other key information, generating a unique MD5 identifier. Upon initial download of a small unit file, a new record is added to a temporary file table to store key information such as the file MD5 and to mark the processing status. After adding the record, the loading operation of file data into the temporary file data table is performed synchronously: the file is split according to agreed-upon newline characters to obtain a list of line data, and the file line data is loaded in batches into the temporary file data table. The temporary file table is queried based on the file MD5 to identify whether duplicate files exist and the current file processing status: for processed files, the processing task flow is adjusted to prevent duplicate job execution; for unprocessed files, duplicate records in the temporary file table are deleted, and associated records of duplicate files in the temporary file data table are simultaneously cleaned up. The latest small unit file information is maintained in the temporary file table, with optimistic concurrency control implemented through version numbers throughout the process. Based on the processing status of the temporary file table, duplicate job execution is filtered. For small unit files that have not yet been processed in the temporary file table, the temporary file data table is queried and complex business processing is performed line by line. At the same time, the processing status of the temporary file data table is updated in real time. When all rows of data in the current file have been processed, the processing status of the file in the temporary file table is updated immediately to associate the synchronization information between the small unit files and the file data, so as to facilitate unified tracking and maintenance.

[0070] For example, based on the MD5 hash algorithm, the control business unit generates a unique file identifier by reading the content of the unit's small file and stores it in a local temporary file table. The business unit judges the status by querying the status field corresponding to the file identifier in the temporary file table. If it is determined that the file is unprocessed, the business unit first loads the unit's small file data into the temporary file data table in batches, and then performs business logic processing (such as data verification, rule calculation, association update, etc.) through a multi-threaded parallel or distributed computing framework. After the processing is completed, the result file is generated, the status of the temporary file table is updated to processed, and the storage path of the result file is recorded.

[0071] In this embodiment, the uniqueness of the MD5 identifier is used to prevent invalid processing of duplicate files from the source. The temporary file table records the processing status in real time to achieve precise filtering of the workflow, while the temporary file data table ensures the traceability of the progress of row data processing.

[0072] In one exemplary embodiment, the method further includes:

[0073] If the processing status is "processed", then skip the execution of business logic processing for the small unit file.

[0074] For example, the control business unit queries the temporary file table using the file's unique identifier (such as MD5). If the matched processing status is "processed", the storage path of the result file associated with that status is directly retrieved without repeating the business logic processing. At the same time, the log information of the skipped processing is recorded, including the file identifier, operation time and result file path, which is convenient for auditing and traceability.

[0075] In this embodiment, by accurately identifying the processed state and directly reusing the result file, the repetitive business processing of small unit files is fundamentally avoided, which greatly reduces redundant calculations and resource consumption and improves the overall batch processing efficiency.

[0076] In one exemplary embodiment, after extracting the file features of the source file, the method further includes:

[0077] The source file is processed based on a preset algorithm to obtain a unique identifier for the source file. Based on the unique identifier, the processing status of the source file is determined. If the processing status is unprocessed, subsequent file feature extraction is performed. If the processing status is processed, the processing of the source file is skipped, and the control unit reads the source file to be split from the file storage server.

[0078] For example, based on the MD5 hash algorithm, the control unit performs hash calculations on the binary content or key metadata (such as filename, size, modification time) of the source file to generate a unique identifier; the processing status of the source file is stored in a public file temporary table, and the public unit determines the status by matching the records in the table with the unique identifier. The status includes unprocessed and processed; if it is determined to be unprocessed, the subsequent file feature deep extraction (such as data row count and business field identification) continues; if it is determined to be processed, the unit directly returns to the source file reading stage of the file storage server to avoid repeated splitting and processing, and records the skip log (including unique identifier, operation time, and status reason).

[0079] In this embodiment, after extracting file features, a unique identifier verification and status judgment mechanism at the source file level is added. By accurately encoding the core information of the file using the MD5 hash algorithm, the rapid identification of duplicate source files is achieved from the source.

[0080] In one embodiment, such as Figure 2As shown, a batch processing method for large files is provided, including: controlling a common unit to read the source files to be split from the file storage server and extracting the file features of the source files. Based on the MD5 hash algorithm, the common unit performs hash calculations on the binary content or key metadata (such as filename, size, modification time) of the source files to generate unique identifiers; the processing status of the source files is stored in a common file temporary table, and the common unit determines the status by matching the records in the table with the unique identifiers, with the status including unprocessed and processed; if it is determined to be unprocessed, the subsequent deep extraction of file features (such as data row count and business field identification) continues; if it is determined to be processed, the process directly returns to the source file reading stage of the file storage server to avoid repeated splitting and processing, and records the skip log (including unique identifier, operation time, and status reason). Based on the size data in the file features, the granularity of the segmentation is dynamically matched according to the principle of "small to small, large to large". The first segmentation splits the source file into quantitative block files of a preset size. Based on the business identification data in the file features, combined with the preset "data sharding - data partitioning - business unit" routing mapping relationship, the quantitative block files are read serially and the business affiliation of each line of data is identified for the second segmentation, and the data of the same business unit is classified and grouped into unit large files. Based on the data row count feature in the file features and the evaluation results of the parallel processing capability of the business unit, the task time, I / O throughput, and resource distribution uniformity are selected as indicators to match an appropriate number of rows for each unit large file, and the third segmentation is performed to split it into unit small files with fixed rows. During the segmentation process, an MD5 unique identifier is generated for each unit small file and recorded in a public file temporary table to achieve full-process traceability. The routing identification information includes business characteristic fields (such as customer number range and business type code) carried in the unit file and the unit belonging mark in the file metadata; the public unit uses the matching algorithm (such as hash matching and range matching) in the global routing service to accurately match the routing identification information with the mapping table; after determining the target business unit, the public unit obtains the cluster load status (CPU utilization, memory usage, number of currently unprocessed tasks) of each business unit in real time, and uses a load balancing strategy (such as round-robin and weighted distribution) to push the unit file to the local storage node or message queue of the target business unit; during the file sending process, a distribution log is generated synchronously to record information such as file identifier, target unit, and sending time, and the distribution status in the public file temporary table is updated.Based on the MD5 hash algorithm, the control business unit generates a unique file identifier by reading the content of a small unit file and stores it in a local temporary file table. The business unit determines the status by querying the status field corresponding to the file identifier in the temporary file table. If it is determined to be unprocessed, the business unit first loads the small unit file data in batches into the temporary file data table, and then performs business logic processing (such as data validation, rule calculation, and association updates) through a multi-threaded parallel or distributed computing framework. After processing, the results are aggregated and generated into a result file, the status of the temporary file table is updated to processed, and the storage path of the result file is recorded. Multiple result files are transferred to the file storage server. If the matched processing status is processed, the storage path of the result file associated with that status is directly retrieved without repeating the business logic processing; at the same time, the log information of skipped processing is recorded, including the file identifier, operation time, and result file path, for easy auditing and traceability. The control common unit reads multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

[0081] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0082] In one exemplary embodiment, such as Figure 3 As shown, a batch processing device for large files is provided, including: a reading module 301, a splitting module 302, a processing module 303, and a merging module 304, wherein:

[0083] The reading module is used to control the common unit to read the source file to be split from the file storage server and extract the file characteristics of the source file;

[0084] The segmentation module is used to control the common units to perform the first, second, and third segments on the source file to be split based on file characteristics, resulting in multiple smaller unit files;

[0085] The processing module is used to control the common unit to distribute multiple unit small files to the corresponding business units based on the preset routing mapping relationship; control the business units to process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server;

[0086] The merging module controls the common unit to read multiple result files from the file storage server, merge the multiple result files, and obtain the business result file.

[0087] In one exemplary embodiment, the segmentation module is further configured to:

[0088] Based on the size data in the file features, the control unit performs a first segmentation on the source file to obtain multiple quantitative block files; based on the business data in the file features and the preset routing mapping relationship, the control unit performs a second segmentation on the multiple quantitative block files to obtain unit large files corresponding to different business units; based on preset evaluation indicators and the resource constraints of different business units, the control unit performs a third segmentation on the unit large files to obtain multiple unit small files.

[0089] In one exemplary embodiment, the processing module is further configured to:

[0090] The control unit reads the routing identification information of multiple unit files; matches the routing identification information with the preset routing mapping relationship to determine the target service unit corresponding to each unit file; and sends the multiple unit files to the corresponding target service unit.

[0091] In one exemplary embodiment, the processing module is further configured to:

[0092] Based on a preset algorithm, the control business unit processes multiple small unit files to obtain a unique file identifier. For any small unit file, the processing status of the small unit file is determined based on the unique file identifier. If the processing status is unprocessed, the business logic processing is performed on the small unit file to obtain the result file.

[0093] In one exemplary embodiment, the processing module is further configured to:

[0094] If the processing status is "processed", then skip the execution of business logic processing for the small unit file.

[0095] In one exemplary embodiment, the segmentation module is further configured to:

[0096] The source file is processed based on a preset algorithm to obtain a unique identifier for the source file. Based on the unique identifier, the processing status of the source file is determined. If the processing status is unprocessed, subsequent file feature extraction is performed. If the processing status is processed, the processing of the source file is skipped, and the control unit reads the source file to be split from the file storage server.

[0097] The modules in the aforementioned batch processing device for large files can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0098] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores source files. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a batch processing method for large files.

[0099] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0100] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0101] The control unit reads the source file to be split from the file storage server and extracts the file characteristics of the source file;

[0102] Based on file characteristics, the control unit performs a first split, a second split, and a third split on the source file to be split, resulting in multiple smaller unit files;

[0103] Based on the preset routing mapping relationship, the control public unit distributes multiple unit small files to the corresponding business units; the control business units process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server;

[0104] The control unit reads multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

[0105] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0106] Based on the size data in the file features, the control unit performs a first segmentation on the source file to obtain multiple quantitative block files; based on the business data in the file features and the preset routing mapping relationship, the control unit performs a second segmentation on the multiple quantitative block files to obtain unit large files corresponding to different business units; based on preset evaluation indicators and the resource constraints of different business units, the control unit performs a third segmentation on the unit large files to obtain multiple unit small files.

[0107] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0108] The control unit reads the routing identification information of multiple unit files; matches the routing identification information with the preset routing mapping relationship to determine the target service unit corresponding to each unit file; and sends the multiple unit files to the corresponding target service unit.

[0109] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0110] Based on a preset algorithm, the control business unit processes multiple small unit files to obtain a unique file identifier. For any small unit file, the processing status of the small unit file is determined based on the unique file identifier. If the processing status is unprocessed, the business logic processing is performed on the small unit file to obtain the result file.

[0111] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0112] If the processing status is "processed", then skip the execution of business logic processing for the small unit file.

[0113] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0114] The source file is processed based on a preset algorithm to obtain a unique identifier for the source file. Based on the unique identifier, the processing status of the source file is determined. If the processing status is unprocessed, subsequent file feature extraction is performed. If the processing status is processed, the processing of the source file is skipped, and the control unit reads the source file to be split from the file storage server.

[0115] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0116] The control unit reads the source file to be split from the file storage server and extracts the file characteristics of the source file;

[0117] Based on file characteristics, the control unit performs a first split, a second split, and a third split on the source file to be split, resulting in multiple smaller unit files;

[0118] Based on the preset routing mapping relationship, the control public unit distributes multiple unit small files to the corresponding business units; the control business units process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server;

[0119] The control unit reads multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

[0120] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0121] Based on the size data in the file features, the control unit performs a first segmentation on the source file to obtain multiple quantitative block files; based on the business data in the file features and the preset routing mapping relationship, the control unit performs a second segmentation on the multiple quantitative block files to obtain unit large files corresponding to different business units; based on preset evaluation indicators and the resource constraints of different business units, the control unit performs a third segmentation on the unit large files to obtain multiple unit small files.

[0122] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0123] The control unit reads the routing identification information of multiple unit files; matches the routing identification information with the preset routing mapping relationship to determine the target service unit corresponding to each unit file; and sends the multiple unit files to the corresponding target service unit.

[0124] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0125] Based on a preset algorithm, the control business unit processes multiple small unit files to obtain a unique file identifier. For any small unit file, the processing status of the small unit file is determined based on the unique file identifier. If the processing status is unprocessed, the business logic processing is performed on the small unit file to obtain the result file.

[0126] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0127] If the processing status is "processed", then skip the execution of business logic processing for the small unit file.

[0128] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0129] The source file is processed based on a preset algorithm to obtain a unique identifier for the source file. Based on the unique identifier, the processing status of the source file is determined. If the processing status is unprocessed, subsequent file feature extraction is performed. If the processing status is processed, the processing of the source file is skipped, and the control unit reads the source file to be split from the file storage server.

[0130] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0131] The control unit reads the source file to be split from the file storage server and extracts the file characteristics of the source file;

[0132] Based on file characteristics, the control unit performs a first split, a second split, and a third split on the source file to be split, resulting in multiple smaller unit files;

[0133] Based on the preset routing mapping relationship, the control public unit distributes multiple unit small files to the corresponding business units; the control business units process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server;

[0134] The control unit reads multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

[0135] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0136] Based on the size data in the file features, the control unit performs a first segmentation on the source file to obtain multiple quantitative block files; based on the business data in the file features and the preset routing mapping relationship, the control unit performs a second segmentation on the multiple quantitative block files to obtain unit large files corresponding to different business units; based on preset evaluation indicators and the resource constraints of different business units, the control unit performs a third segmentation on the unit large files to obtain multiple unit small files.

[0137] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0138] The control unit reads the routing identification information of multiple unit files; matches the routing identification information with the preset routing mapping relationship to determine the target service unit corresponding to each unit file; and sends the multiple unit files to the corresponding target service unit.

[0139] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0140] Based on a preset algorithm, the control business unit processes multiple small unit files to obtain a unique file identifier. For any small unit file, the processing status of the small unit file is determined based on the unique file identifier. If the processing status is unprocessed, the business logic processing is performed on the small unit file to obtain the result file.

[0141] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0142] If the processing status is "processed", then skip the execution of business logic processing for the small unit file.

[0143] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0144] The source file is processed based on a preset algorithm to obtain a unique identifier for the source file. Based on the unique identifier, the processing status of the source file is determined. If the processing status is unprocessed, subsequent file feature extraction is performed. If the processing status is processed, the processing of the source file is skipped, and the control unit reads the source file to be split from the file storage server.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A batch processing method for large files, characterized in that, The method includes: The control unit reads the source file to be split from the file storage server and extracts the file characteristics of the source file; Based on the file characteristics, the control common unit performs a first segmentation, a second segmentation, and a third segmentation on the source file to be split, resulting in multiple unit small files; Based on a preset routing mapping relationship, the control public unit distributes the multiple unit small files to the corresponding business units; the control business units process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server; The control unit reads the multiple result files from the file storage server, merges the multiple result files, and obtains the business result file.

2. The method according to claim 1, characterized in that, Based on the file characteristics, the control unit performs a first segmentation, a second segmentation, and a third segmentation on the source file to be split, resulting in multiple smaller unit files, including: Based on the size data in the file features, the control unit performs a first segmentation on the source file to obtain multiple quantitative block files; Based on the business data in the file features and the preset routing mapping relationship, the control public unit performs a second segmentation on the multiple quantitative block files to obtain unit large files corresponding to different business units; Based on preset evaluation indicators and the resource constraints of different business units, the control public unit performs a third segmentation on the large unit file to obtain multiple small unit files.

3. The method according to claim 1, characterized in that, The method of controlling the public unit to distribute the multiple unit small files to the corresponding business units based on the preset routing mapping relationship includes: The control unit reads the routing identifier information of the multiple unit small files; The routing identification information is matched with the preset routing mapping relationship to determine the target service unit corresponding to each unit file; Send multiple small unit files to the corresponding target business unit.

4. The method according to claim 1, characterized in that, The control unit processes the multiple small files to obtain multiple result files, including: Based on a preset algorithm, the business unit is controlled to process the multiple small files to obtain a unique file identifier; For any unit file, the processing status of the unit file is determined based on the file's unique identifier; If the processing status is "unprocessed", then business logic processing is performed on the unit file to obtain the result file.

5. The method according to claim 4, characterized in that, The method further includes: If the processing status is "processed", then skip the execution of business logic processing for the unit file.

6. The method according to claim 1, characterized in that, After extracting the file features of the source file, the process further includes: The source file is processed based on a preset algorithm to obtain a unique identifier for the source file; Based on the unique identifier, determine the processing status of the source file; If the processing status is unprocessed, then proceed with subsequent file feature extraction; If the processing status is "processed", then skip the processing of the source file and return to the control unit to read the source file to be split from the file storage server.

7. A batch processing device for large files, characterized in that, The device includes: The reading module is used to control the common unit to read the source file to be split from the file storage server and extract the file characteristics of the source file; The segmentation module is used to control the common unit to perform a first segmentation, a second segmentation, and a third segmentation on the source file to be split based on the file characteristics, so as to obtain multiple unit small files; The processing module is used to control the public unit to distribute the multiple unit small files to the corresponding business units based on a preset routing mapping relationship; control the business units to process the multiple unit small files to obtain multiple result files, and transmit the multiple result files to the file storage server; The merging module is used to control the common unit to read the multiple result files from the file storage server, merge the multiple result files, and obtain the business result file.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images