Large file batch processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a large file batch processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a to-be-processed source file and performing virtual fragmentation processing on the source file to obtain a plurality of fragmentation results; sending the plurality of fragmentation results to a plurality of processing nodes; for any processing node, reading data in the fragmentation result through a preset reader component to obtain a plurality of byte arrays; performing service logic processing on the byte array through a preset processor component to obtain a processing result; writing the processing result into a temporary data file through a preset writer component; the processing results in the temporary data file are written into a database, and whether all the processing results are successfully written into the database or not is judged; and if all the processing results are successfully written into the database, deleting the temporary data file. By adopting the method, the data importing efficiency of the large file can be improved, and the consistency of data processing and the high availability of the system are guaranteed.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for batch processing of large files. Background Technology

[0002] With the standardization of the financial market, the China Securities Regulatory Commission (CSRC) issued the "Interim Provisions on the Management of Settlement Funds for Securities Investment Fund Sales," which explicitly requires commercial banks, as supervisory bodies, to establish a supervisory information platform connected to the sales business information management platform of account holders. This platform is responsible for core supervisory functions such as fund investor identity authentication, account association, fund transfer control, fund flow monitoring, large-amount early warning processing, and fund sales data import. It is a key infrastructure for ensuring the safety of fund sales settlement funds and regulating market order. In actual business scenarios, the supervisory information platform needs to receive and process thousands of batches of transaction details, investor information files, and investor account information files from fund sales institutions daily, as well as various data files such as transaction confirmations and dividends from registration and settlement institutions. The size of a single file can reach several hundred gigabytes, and the scale and complexity of data processing continue to rise.

[0003] Existing methods are insufficient to meet the timeliness requirements of business data entry, and are prone to problems such as processing timeouts and loss of data consistency, which in turn affect the timely implementation of supervisory functions and make it impossible to respond efficiently to regulatory and business needs. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for batch processing of large files that can improve data import efficiency while ensuring data consistency and high system availability, in order to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for batch processing large files, including:

[0006] Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes;

[0007] For any processing node, the data in the sharding result is read by the preset reader component to obtain multiple byte arrays; the byte arrays are processed by the preset processor component to obtain the processing result; and the processing result is written to a temporary data file by the preset writer component.

[0008] Write the processing results from the temporary data file into the database, and check whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0009] In one embodiment, data from the fragmented results is read using a preset reader component to obtain multiple byte arrays, including:

[0010] The preset reader component determines the start and end byte positions of the fragmented result in the source file; the preset reader component reads data starting from the start byte position of the source file, and obtains a set of byte data each time a newline character is read, until the end byte position of the source file is read, obtaining multiple sets of byte data.

[0011] In one embodiment, a pre-defined processor component performs business logic processing on the byte array to obtain a processing result, including:

[0012] The type of the byte array is determined, and based on the type, the target processor in the preset processor components is determined; the byte array is cleaned, validated, and processed by the target processor to obtain the processing result.

[0013] In one embodiment, the processing result is written to a temporary data file via a preset writer component, including:

[0014] Whenever the number of processing results reaches a preset number, the preset number of processing results are concatenated into a large byte array through a preset writer component; the large byte array is then written to a temporary data file; if the writing fails, it is determined whether the number of times the large byte array has been written to the temporary data file exceeds a preset number; if it has not exceeded the preset number, the process of writing the large byte array to the temporary data file is returned; if it exceeds the preset number, the processing status of the sharding result is "failed to be inserted into the database".

[0015] In one embodiment, the method further includes:

[0016] If any processing result fails to be written to the database, the processing status of the corresponding shard result will be changed to "failed to write to database".

[0017] In one embodiment, the method further includes:

[0018] Every preset time interval, the processing status of all shard results corresponding to the source file to be processed is obtained; if the processing status of any shard result is "failed to be entered into the database", the source file to be processed fails to be entered into the database, a failure report corresponding to the source file to be processed is generated, and the failure report is sent to the operation and maintenance platform.

[0019] Secondly, this application also provides a large file batch processing apparatus, comprising:

[0020] The acquisition module is used to acquire the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; and send the multiple fragmentation results to multiple processing nodes.

[0021] The processing module is used to read data from the sharding results for any processing node through a preset reader component to obtain multiple byte arrays; to perform business logic processing on the byte arrays through a preset processor component to obtain the processing result; and to write the processing result to a temporary data file through a preset writer component.

[0022] The write module is used to write the processing results from the temporary data file into the database and to determine whether all processing results have been successfully written to the database; if all processing results have been successfully written to the database, the temporary data file is deleted.

[0023] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0024] Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes;

[0025] For any processing node, the data in the sharding result is read by the preset reader component to obtain multiple byte arrays; the byte arrays are processed by the preset processor component to obtain the processing result; and the processing result is written to a temporary data file by the preset writer component.

[0026] Write the processing results from the temporary data file into the database, and check whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0027] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0028] Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes;

[0029] For any processing node, the data in the sharding result is read by the preset reader component to obtain multiple byte arrays; the byte arrays are processed by the preset processor component to obtain the processing result; and the processing result is written to a temporary data file by the preset writer component.

[0030] Write the processing results from the temporary data file into the database, and check whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0031] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0032] Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes;

[0033] For any processing node, the data in the sharding result is read by the preset reader component to obtain multiple byte arrays; the byte arrays are processed by the preset processor component to obtain the processing result; and the processing result is written to a temporary data file by the preset writer component.

[0034] Write the processing results from the temporary data file into the database, and check whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0035] The aforementioned large file batch processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product first acquire the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; these multiple fragmentation results are then sent to multiple processing nodes; next, for any processing node, a preset reader component reads the data from the fragmentation results to obtain multiple byte arrays; a preset processor component performs business logic processing on the byte arrays to obtain processing results; a preset writer component writes the processing results to a temporary data file; finally, the processing results in the temporary data file are written to a database, and it is determined whether all processing results have been successfully written to the database; if all processing results have been successfully written to the database, the temporary data file is deleted. Virtual fragmentation avoids the resource consumption of physical splitting, byte-level processing components eliminate redundant data type conversions, and multi-node distributed parallel processing overcomes single-machine performance bottlenecks. Combined with database batch loading optimization and transaction management mechanisms, this significantly improves the efficiency of large file data import, ensures data processing consistency and system high availability, and effectively meets the timeliness requirements of business operations in large-scale data scenarios. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating a large file batch processing method in one embodiment;

[0038] Figure 2 This is a detailed flowchart of a large file batch processing method in one embodiment;

[0039] Figure 3 This is a structural block diagram of a large file batch processing device in one embodiment;

[0040] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] In one embodiment, such as Figure 1 As shown, a method for batch processing large files is provided. This embodiment illustrates the method using a terminal as an example. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0043] Step 102: Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes.

[0044] For example, obtaining the source files to be processed includes: periodically scanning the shared storage directory or receiving external file push notifications to obtain transaction detail files, investor information files, transaction confirmation files, and other files awaiting processing uploaded by fund sales institutions and registration agencies; recording file metadata (including filename, total size, creation time, character set type, and MD5 checksum); and verifying file integrity by comparing the MD5 value. After successful verification, the file is locked to prevent duplicate processing. Virtual sharding processing includes: calculating the total number of shards based on the preset single shard byte size (dynamically configurable) and the total number of bytes in the file; determining the start and end byte positions for each shard; if the end byte position of a shard falls in the middle of a data line, marking the shard with a "boundary adjustment flag"; if it is the first shard, marking it with a "file header extraction flag"; and generating a sharding result containing shard number, filename, start / end byte position, and flag information. No physical file splitting is performed throughout the process. Sending multiple sharded results to multiple processing nodes includes: distributing sharded tasks through Kafka message queues, encapsulating each sharded result into a message containing core task information, and publishing it to a specified Kafka topic; multiple processing nodes listen to the topic as members of the same consumer group, and Kafka automatically distributes messages evenly to idle nodes based on node load status. After receiving the message, the node parses the sharding information and marks the task processing status.

[0045] Step 104: For any processing node, read the data in the sharding result through the preset reader component to obtain multiple byte arrays; perform business logic processing on the byte arrays through the preset processor component to obtain the processing result; and write the processing result to a temporary data file through the preset writer component.

[0046] For example, the data in the sharding result is read through a preset reader component to obtain multiple sets of byte arrays, including: the processing node calls a custom ByteArrayFlatFileItemReader component, which, according to the sharding result, starts a BufferedInputStream to read the file one by one and writes the bytes to a ByteArrayOutputStream; if the sharding result contains a "boundary adjustment flag", the incomplete data in the first line is discarded; if it contains a "file header flag", the file header byte array is parsed and stored in the execution context; character set conversion is performed on the byte array (if the source file and database character sets are inconsistent), and the data is read in a loop until the end byte position is reached, finally obtaining multiple sets of business data byte arrays. The business logic processing of the byte array through the preset processor component includes: calling a custom ItemProcessor component that matches the file type to perform data cleaning (removing invalid bytes, correcting format errors), data filtering (filtering valid byte arrays according to business rules), data validation (verifying the byte length and value range of key fields), and data processing (appending flag bytes, adjusting field order), etc., at the byte level, without converting to strings or entity objects. Data that fails validation is written to the local exception log, and a new byte array is output as the processing result after successful processing. The pre-defined writer component writes the processing results to a temporary data file, including calling a custom ByteArrayFlatFileItemWriter component to write the local temporary data file of the node in batches (the file path includes the shard number for differentiation). After each batch of writing is completed, a transaction commit is triggered. In case of an exception, rollback and retry are performed to ensure the integrity of the data writing.

[0047] Step 106: Write the processing results from the temporary data file into the database, and determine whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0048] For example, writing the processing results from the temporary data file into the database includes: the processing node automatically generates an SQLLoader control file based on the temporary data file, configuring field separators, data type mappings, and error handling rules; enabling the database direct path loading mode and batch commit optimization strategy, controlling the number of concurrent loads per table through semaphores, and executing the sqlldr command to batch load the byte stream data from the temporary data file into the corresponding table in the Oracle database; monitoring the loading progress and error rate in real time during the loading process, and terminating the loading and triggering an alarm if the error rate exceeds the threshold. Determining whether all processing results have been successfully written to the database includes: the master node summarizing the shard loading status of each processing node, verifying whether the number of loaded records for each shard is consistent with the number of rows of shard data in the source file, and checking whether the MD5 checksum of the data in the database table matches the checksum of the temporary file data; if all shards have been loaded and the data consistency check passes, then all processing results are considered to have been successfully written to the database; if there are shard loading failures or data inconsistencies, then abnormal shards are marked and a retry mechanism is triggered; if all processing results have been successfully written to the database, deleting temporary data files includes: after each processing node receives the successful confirmation instruction from the master node, it deletes the local temporary data files and the SQLLoader control file, releasing the disk space occupied by the files; the master node synchronously deletes the shard task metadata records, archives the source files to the historical storage directory, and updates the file processing status to "completed"; if there are shards that have failed to load, then the temporary files and error logs are retained for maintenance and troubleshooting, and the loading process is re-executed after the problem is resolved.

[0049] The aforementioned large file batch processing method first obtains the source file to be processed and performs virtual sharding on it, resulting in multiple sharded results. These sharded results are then sent to multiple processing nodes. Next, for each processing node, a pre-defined reader component reads the data from the sharded results, obtaining multiple byte arrays. A pre-defined processor component performs business logic processing on these byte arrays to obtain the processing results. A pre-defined writer component writes the processing results to a temporary data file. Finally, the processing results in the temporary data file are written to the database, and it is determined whether all processing results have been successfully written to the database. If all processing results have been successfully written to the database, the temporary data file is deleted. By avoiding the resource consumption of physical splitting through virtual sharding, eliminating redundant data type conversions through byte-level processing components, and overcoming single-machine performance bottlenecks through multi-node distributed parallel processing, combined with database batch loading optimization and transaction management mechanisms, this method significantly improves the efficiency of large file data import, ensures data processing consistency and system high availability, and effectively meets the timeliness requirements of business operations in large-scale data scenarios.

[0050] In an exemplary embodiment, data from the fragmented results is read using a preset reader component to obtain multiple byte arrays, including:

[0051] Determine the start byte position and end byte position of the sharding result in the source file through a preset reader component; start reading data from the start byte position of the source file through the preset reader component. Whenever a line break is read, a set of byte data is obtained until the end byte position of the source file is reached, and multiple sets of byte data are obtained.

[0052] Exemplarily, assume that the total size of a certain fund transaction details file is 50GB. After virtual sharding by 10GB per shard, the start byte position of the second shard is 10737418240 and the end byte position is 21474836480; the reader component jumps to the 10737418240th byte through the skip() method of FileInputStream, starts reading byte by byte through BufferedInputStream, and generates a byte array of the complete data of that line when the first line break is read (such as the byte array corresponding to the transaction record "001|20251124|purchase|10000"), and then continues to read backward. Each time a line break is encountered, a new set of byte data is generated until the reading position reaches the 21474836480th byte, and finally tens of thousands of sets of transaction data byte arrays included in this shard are obtained; if the start byte position of the second shard exactly falls in the middle of a certain line of data (such as after "001|20251124|purchase"), the reader component will discard the incomplete data of that line and start generating byte arrays from the next complete line of data to ensure data integrity.

[0053] In this embodiment, the reader component uses a custom ByteArrayFlatFileItemReader component to process data in the form of a byte stream throughout the process, without converting the byte array into a string or an entity object, reducing performance loss from the source. Accurately jump to the start byte position of the shard through the skip() method of FileInputStream, avoid reading irrelevant preamble data, and improve the reading efficiency; read byte by byte through BufferedInputStream and cache through ByteArrayOutputStream to ensure the complete splicing of each line of data, and output a set of byte arrays until a line break (\r, \n or \r\n) is detected, realizing the accurate splitting of line-level data.

[0054] In an exemplary embodiment, perform business logic processing on the byte array through a preset processor component to obtain a processing result, including:

[0055] Determine the type of the byte array, and based on the type, determine the target processor in the preset processor component; perform data cleaning, data verification, and data processing on the byte array through the target processor to obtain a processing result.

[0056] For example, assuming the read byte array corresponds to fund investor information file data, and its type is determined to be "Investor Information", then the target processor is matched from the preset processor components as InvestorInfoItemProcessor. This target processor first performs data cleaning, removing redundant spaces, special characters, and other invalid bytes from the byte array, and correcting format errors such as mismatched field lengths. Then, it performs data validation, verifying whether the byte length corresponding to the ID number field conforms to the 18-bit standard and whether the mobile phone number field bytes are purely numeric. If the validation fails, an exception log is recorded. Finally, it performs data processing, appending data source identifier bytes to the end of the byte array and adjusting the field order (such as moving the name field byte array to the first position), and finally outputting a standardized processed result byte array.

[0057] In this embodiment, the preset processor component includes multiple target processors, which are adapted to different types of file data such as transaction details, investor information, and transaction confirmation. The file type tags corresponding to the byte array are quickly matched, eliminating the need for a general-purpose processor to be compatible with multiple scenarios and improving processing efficiency.

[0058] In an exemplary embodiment, the processing result is written to a temporary data file via a preset writer component, including:

[0059] Whenever the number of processing results reaches a preset number, the preset number of processing results are concatenated into a large byte array through a preset writer component; the large byte array is then written to a temporary data file; if the writing fails, it is determined whether the number of times the large byte array has been written to the temporary data file exceeds a preset number; if it has not exceeded the preset number, the process of writing the large byte array to the temporary data file is returned; if it exceeds the preset number, the processing status of the sharding result is "failed to be inserted into the database".

[0060] For example, assuming a preset quantity of 1000 processing results, when the cumulative number of investor information processing results output by the target processor reaches 1000 sets of byte arrays, the writer component calls the System.arraycopy method to merge these 1000 sets of byte arrays into a single continuous large byte array according to the rule of "field separator + newline character" (e.g., each set of byte arrays is separated by "|", and groups are separated by "\n"). Then, this large byte array is written in batches to a temporary file on the node's local machine. If a write failure occurs due to disk I / O anomalies, the writer component automatically records the number of failures and immediately retryes, with a preset retry count of 3. If the third retry still fails, the processing status of the fragment result is marked as "database entry failed," and an alarm is triggered to notify maintenance personnel to check the disk status.

[0061] In this embodiment, the writer component uses a custom ByteArrayFlatFileItemWriter component to replace line-by-line writing by batch concatenating large byte arrays, significantly reducing the number of disk I / O operations and improving writing efficiency. The preset number can be dynamically adjusted according to server performance (e.g., 2000 records for high-performance nodes and 500 records for low-performance nodes), balancing writing efficiency and memory usage. The retry mechanism effectively handles transient I / O failures, ensuring the reliability of data writing. The real-time marking of the shard processing status allows the master node to promptly grasp the progress of each shard and trigger a reprocessing process for failed shards, avoiding the overall task from stalling due to a single shard failure, thus adapting to the high availability requirements of large-scale file processing.

[0062] In one exemplary embodiment, the method further includes:

[0063] If any processing result fails to be written to the database, the processing status of the corresponding shard result will be changed to "failed to write to database".

[0064] For example, when the third shard of a fund transaction details file is loaded into an Oracle database in batches via SQLLoader, the loading is interrupted due to insufficient database tablespace. After the processing node detects the loading failure signal, it immediately changes the processing status of the shard in the task management table from "loading" to "failed to load", and records the reason for failure as "insufficient tablespace, number of rows loaded 15600 / 89200". At the same time, it triggers the alarm mechanism and pushes an alarm notification containing shard information and the reason for failure to the operation and maintenance system for timely handling.

[0065] In this embodiment, the system synchronizes the processing status of each shard in real time through a distributed task status management table. When a processing node detects a database write failure (such as connection timeout, insufficient tablespace, constraint conflicts, etc.), it atomically updates the shard status and records the failure details, ensuring the consistency and traceability of status information. This design enables the master node to quickly identify abnormal shards and automatically trigger a retry mechanism (such as reloading after tablespace expansion) for shards that have failed to write data, or mark them as requiring manual intervention. This avoids blocking the entire data processing chain due to the failure of a single shard, ensuring the stability and fault tolerance of large-scale file batch processing tasks.

[0066] In one exemplary embodiment, the method further includes:

[0067] Every preset time interval, the processing status of all shard results corresponding to the source file to be processed is obtained; if the processing status of any shard result is "failed to be entered into the database", the source file to be processed fails to be entered into the database, a failure report corresponding to the source file to be processed is generated, and the failure report is sent to the operation and maintenance platform.

[0068] For example, the preset time period is set to 1 minute. Every minute, the processing status of all shard results is scanned. If a certain shard status is found to be "failed to be inserted into the database" (failure reason: database connection timeout), the master node will then determine that the entire source file has failed to be inserted into the database, automatically generate a failure report, and the report includes information such as file name, total number of shards, failed shard sequence number, failure reason, and failure time. The report is pushed to the operation and maintenance monitoring platform through the HTTP interface, and at the same time, the platform alarm pop-up and SMS notification are triggered.

[0069] In this embodiment, the master node globally controls the sharding processing status of source files through a timed polling mechanism. The preset time period can be flexibly adjusted according to business timeliness requirements (e.g., 1 minute for core transaction files and 10 minutes for non-core information files). Failure reports not only include shard-level failure details but also associate source file metadata (such as file size, uploading institution, and data batch), facilitating rapid problem identification by operations and maintenance personnel. Report pushes utilize multiple channels (platform messages, SMS, and email) to ensure timely response to anomalies. This mechanism achieves linked judgment from sharding anomalies to the overall file status, preventing data omissions due to undetected individual shard failures and ensuring the integrity and regulatory compliance of fund supervision data.

[0070] In one embodiment, such as Figure 2As shown, a large file batch processing method is provided, including: obtaining the source file to be processed and performing virtual sharding on the source file to obtain multiple sharding results; sending the multiple sharding results to multiple processing nodes. For any processing node, assuming the total size of a certain fund transaction detail file is 50GB, after virtual sharding by 10GB per shard, the starting byte position of the second shard is 10737418240 and the ending byte position is 21474836480; the ByteArrayFlatFileItemReader (reader) component jumps to the 10737418240 - byte position through the skip() method of FileInputStream, starts to read byte by byte using BufferedInputStream, generates a byte array of the complete data of the line when the first line feed character is read (such as the byte array corresponding to the transaction record "001|20251124|purchase|10000"), then continues to read backward, generates a new set of byte data every time a line feed character is encountered until the reading position reaches 21474836480 bytes, and finally obtains tens of thousands of groups of transaction data byte arrays included in this shard; if the starting byte position of the second shard exactly falls in the middle of a certain line of data (such as after "001|20251124|purchase"), the reader component will discard the incomplete data of this line and start generating byte arrays from the next complete line to ensure data integrity. Assuming the byte array read corresponds to the data of the fund investor information file and its type is determined to be "investor information type", the target processor is matched as InvestorInfoItemProcessor from the preset processor components; the target processor first performs data cleaning, removing invalid bytes such as extra spaces and special characters in the byte array and correcting format errors with mismatched field lengths; then performs data verification, verifying whether the byte length corresponding to the ID card number field meets the 18 - digit standard and whether the byte of the mobile phone number field is pure digital encoding, and records exception logs if the verification fails; finally, performs data processing, appending a data source identification byte at the end of the byte array and adjusting the field order (such as moving the byte array of the name field to the first place), and finally outputs a standardized processed result byte array. Assuming the preset quantity is 1000 processing results, when the cumulative number of investor information processing results output by the target processor reaches 1000 groups of byte arrays, the writer component calls the System.arraycopy method to merge these 1000 groups of byte arrays into a continuous large byte array according to the rule of "field separator + line feed character" (such as each group of byte arrays is separated by "|" and the groups are separated by "\n"); then writes this large byte array in batches to the temporary file on the local node of the node. If the write fails due to disk I / O exception, the writer component automatically records the number of failures and immediately retries, with the preset number of retries being 3 times; if the 3rd retry still fails, the processing status of this shard result is marked as "failed to be stored in the warehouse", and an alarm is triggered to notify the operation and maintenance personnel to check the disk status.The processing results in the temporary data file are written to the database, and it is determined whether all processing results have been successfully written to the database. If all processing results have been successfully written to the database, the temporary data file is deleted. If any processing result fails to be written to the database, the processing status of the corresponding shard result is changed to "database insertion failure". The processing status of all shard results is scanned every minute. If a shard status is found to be "database insertion failure" (failure reason: database connection timeout), the master node immediately determines that the entire source file has failed to be inserted into the database, automatically generates a failure report, and the report includes information such as file name, total number of shards, failed shard sequence number, failure reason, and failure time. The report is pushed to the operation and maintenance monitoring platform via HTTP interface, and at the same time, the platform alarm pop-up and SMS notification are triggered.

[0071] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0072] In one exemplary embodiment, such as Figure 3 As shown, a large file batch processing device is provided, including: an acquisition module 301, a processing module 302, and a writing module 303, wherein:

[0073] The acquisition module is used to acquire the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; and send the multiple fragmentation results to multiple processing nodes.

[0074] The processing module is used to read data from the sharding results for any processing node through a preset reader component to obtain multiple byte arrays; to perform business logic processing on the byte arrays through a preset processor component to obtain the processing result; and to write the processing result to a temporary data file through a preset writer component.

[0075] The write module is used to write the processing results from the temporary data file into the database and to determine whether all processing results have been successfully written to the database; if all processing results have been successfully written to the database, the temporary data file is deleted.

[0076] In one exemplary embodiment, the processing module is further configured to:

[0077] The preset reader component determines the start and end byte positions of the fragmented result in the source file; the preset reader component reads data starting from the start byte position of the source file, and obtains a set of byte data each time a newline character is read, until the end byte position of the source file is read, obtaining multiple sets of byte data.

[0078] In one exemplary embodiment, the processing module is further configured to:

[0079] The type of the byte array is determined, and based on the type, the target processor in the preset processor components is determined; the byte array is cleaned, validated, and processed by the target processor to obtain the processing result.

[0080] In one exemplary embodiment, the processing module is further configured to:

[0081] Whenever the number of processing results reaches a preset number, the preset number of processing results are concatenated into a large byte array through a preset writer component; the large byte array is then written to a temporary data file; if the writing fails, it is determined whether the number of times the large byte array has been written to the temporary data file exceeds a preset number; if it has not exceeded the preset number, the process of writing the large byte array to the temporary data file is returned; if it exceeds the preset number, the processing status of the sharding result is "failed to be inserted into the database".

[0082] In one exemplary embodiment, the processing module is further configured to:

[0083] If any processing result fails to be written to the database, the processing status of the corresponding shard result will be changed to "failed to write to database".

[0084] In one exemplary embodiment, the processing module is further configured to:

[0085] Every preset time interval, the processing status of all shard results corresponding to the source file to be processed is obtained; if the processing status of any shard result is "failed to be entered into the database", the source file to be processed fails to be entered into the database, a failure report corresponding to the source file to be processed is generated, and the failure report is sent to the operation and maintenance platform.

[0086] Each module in the aforementioned large file batch processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0087] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores source files. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a large file batch processing method.

[0088] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0089] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0090] Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes;

[0091] For any processing node, the data in the sharding result is read by the preset reader component to obtain multiple byte arrays; the byte arrays are processed by the preset processor component to obtain the processing result; and the processing result is written to a temporary data file by the preset writer component.

[0092] Write the processing results from the temporary data file into the database, and check whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0093] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0094] The preset reader component determines the start and end byte positions of the fragmented result in the source file; the preset reader component reads data starting from the start byte position of the source file, and obtains a set of byte data each time a newline character is read, until the end byte position of the source file is read, obtaining multiple sets of byte data.

[0095] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0096] The type of the byte array is determined, and based on the type, the target processor in the preset processor components is determined; the byte array is cleaned, validated, and processed by the target processor to obtain the processing result.

[0097] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0098] Whenever the number of processing results reaches a preset number, the preset number of processing results are concatenated into a large byte array through a preset writer component; the large byte array is then written to a temporary data file; if the writing fails, it is determined whether the number of times the large byte array has been written to the temporary data file exceeds a preset number; if it has not exceeded the preset number, the process of writing the large byte array to the temporary data file is returned; if it exceeds the preset number, the processing status of the sharding result is "failed to be inserted into the database".

[0099] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0100] If any processing result fails to be written to the database, the processing status of the corresponding shard result will be changed to "failed to write to database".

[0101] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0102] Every preset time interval, the processing status of all shard results corresponding to the source file to be processed is obtained; if the processing status of any shard result is "failed to be entered into the database", the source file to be processed fails to be entered into the database, a failure report corresponding to the source file to be processed is generated, and the failure report is sent to the operation and maintenance platform.

[0103] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0104] Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes;

[0105] For any processing node, the data in the sharding result is read by the preset reader component to obtain multiple byte arrays; the byte arrays are processed by the preset processor component to obtain the processing result; and the processing result is written to a temporary data file by the preset writer component.

[0106] Write the processing results from the temporary data file into the database, and check whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0107] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0108] The preset reader component determines the start and end byte positions of the fragmented result in the source file; the preset reader component reads data starting from the start byte position of the source file, and obtains a set of byte data each time a newline character is read, until the end byte position of the source file is read, obtaining multiple sets of byte data.

[0109] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0110] The type of the byte array is determined, and based on the type, the target processor in the preset processor components is determined; the byte array is cleaned, validated, and processed by the target processor to obtain the processing result.

[0111] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0112] Whenever the number of processing results reaches a preset number, the preset number of processing results are concatenated into a large byte array through a preset writer component; the large byte array is then written to a temporary data file; if the writing fails, it is determined whether the number of times the large byte array has been written to the temporary data file exceeds a preset number; if it has not exceeded the preset number, the process of writing the large byte array to the temporary data file is returned; if it exceeds the preset number, the processing status of the sharding result is "failed to be inserted into the database".

[0113] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0114] If any processing result fails to be written to the database, the processing status of the corresponding shard result will be changed to "failed to write to database".

[0115] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0116] Every preset time interval, the processing status of all shard results corresponding to the source file to be processed is obtained; if the processing status of any shard result is "failed to be entered into the database", the source file to be processed fails to be entered into the database, a failure report corresponding to the source file to be processed is generated, and the failure report is sent to the operation and maintenance platform.

[0117] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0118] Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes;

[0119] For any processing node, the data in the sharding result is read by the preset reader component to obtain multiple byte arrays; the byte arrays are processed by the preset processor component to obtain the processing result; and the processing result is written to a temporary data file by the preset writer component.

[0120] Write the processing results from the temporary data file into the database, and check whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, delete the temporary data file.

[0121] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0122] The preset reader component determines the start and end byte positions of the fragmented result in the source file; the preset reader component reads data starting from the start byte position of the source file, and obtains a set of byte data each time a newline character is read, until the end byte position of the source file is read, obtaining multiple sets of byte data.

[0123] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0124] The type of the byte array is determined, and based on the type, the target processor in the preset processor components is determined; the byte array is cleaned, validated, and processed by the target processor to obtain the processing result.

[0125] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0126] Whenever the number of processing results reaches a preset number, the preset number of processing results are concatenated into a large byte array through a preset writer component; the large byte array is then written to a temporary data file; if the writing fails, it is determined whether the number of times the large byte array has been written to the temporary data file exceeds a preset number; if it has not exceeded the preset number, the process of writing the large byte array to the temporary data file is returned; if it exceeds the preset number, the processing status of the sharding result is "failed to be inserted into the database".

[0127] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0128] If any processing result fails to be written to the database, the processing status of the corresponding shard result will be changed to "failed to write to database".

[0129] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0130] Every preset time interval, the processing status of all shard results corresponding to the source file to be processed is obtained; if the processing status of any shard result is "failed to be entered into the database", the source file to be processed fails to be entered into the database, a failure report corresponding to the source file to be processed is generated, and the failure report is sent to the operation and maintenance platform.

[0131] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0132] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for batch processing large files, characterized in that, The method includes: Obtain the source file to be processed and perform virtual fragmentation on the source file to obtain multiple fragmentation results; send the multiple fragmentation results to multiple processing nodes; For any processing node, the data in the sharding result is read by a preset reader component to obtain multiple byte arrays; the byte arrays are processed by a preset processor component to obtain the processing result; and the processing result is written to a temporary data file by a preset writer component. The processing results in the temporary data file are written to the database, and it is determined whether all processing results have been successfully written to the database; if all processing results have been successfully written to the database, the temporary data file is deleted.

2. The method according to claim 1, characterized in that, The process involves reading data from the fragmented results using a preset reader component to obtain multiple byte arrays, including: The start and end byte positions of the fragmented result in the source file are determined by a preset reader component. The preset reader component reads data starting from the beginning byte position of the source file. Each time a newline character is read, a set of bytes is obtained until the end byte position of the source file is read, resulting in multiple sets of bytes.

3. The method according to claim 1, characterized in that, The step of performing business logic processing on the byte array through a preset processor component to obtain the processing result includes: Determine the type of the byte array, and based on the type, determine the target processor in the preset processor component; The target processor performs data cleaning, data verification, and data processing on the byte array to obtain the processing result.

4. The method according to claim 1, characterized in that, The step of writing the processing result to a temporary data file using a preset writer component includes: Whenever the number of processing results reaches a preset number, the preset number of processing results are concatenated into a large byte array through a preset writer component; Write the large byte array to a temporary data file; If the write operation fails, determine whether the number of times the large byte array has been written to the temporary data file exceeds a preset number. If the preset number of attempts is not exceeded, return and write the large byte array to a temporary data file; If the preset number of attempts is exceeded, the processing status of the sharding result will be "database entry failed".

5. The method according to claim 1, characterized in that, The method further includes: If any processing result fails to be written to the database, the processing status of the shard result corresponding to that processing result will be changed to "failed to write to database".

6. The method according to claim 1, characterized in that, The method further includes: Every preset time interval, obtain the processing status of all the fragment results corresponding to the source file to be processed; If any shard result has a processing status of "failed to be inserted into the database", then the source file to be processed has failed to be inserted into the database. A failure report corresponding to the source file to be processed is generated and the failure report is sent to the operation and maintenance platform.

7. A large file batch processing device, characterized in that, The device includes: The acquisition module is used to acquire the source file to be processed and perform virtual fragmentation processing on the source file to obtain multiple fragmentation results; and send the multiple fragmentation results to multiple processing nodes. The processing module is used to read data from the sharding result through a preset reader component for any processing node to obtain multiple byte arrays; to perform business logic processing on the byte arrays through a preset processor component to obtain processing results; and to write the processing results into a temporary data file through a preset writer component. The writing module is used to write the processing results in the temporary data file into the database and determine whether all processing results have been successfully written into the database; if all processing results have been successfully written into the database, the temporary data file is deleted.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images