Large file batch processing method and device, computer equipment, readable storage medium and program product

By using virtual sharding and component-based data processing, the timeliness and consistency issues of large-scale fund transaction data processing were resolved, achieving efficient data import and system stability.

CN121833622APending Publication Date: 2026-04-10SHANGHAI PUDONG DEVELOPMENT BANK
0 Cites 0 Cited by

Patent Information

Application Number
CN202511802902.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-02
Publication Date
2026-04-10

AI Technical Summary

Technical Problem

Existing methods are insufficient to meet the timeliness requirements of financial market supervision information platforms in processing large-scale fund transaction data. They are prone to problems such as processing timeouts and data inconsistency, which affect the timely implementation of supervisory functions.

Method used

A virtual sharding processing method is adopted to send the sharded results of large files to multiple processing nodes. Data processing and writing are performed through reader, processor and writer components. Combined with database batch loading optimization and transaction management mechanism, data consistency and system high availability are ensured.

Benefits of technology

It significantly improves the efficiency of importing large file data, ensures the consistency of data processing and high availability of the system, and meets the timeliness requirements of business in large-scale data scenarios.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention relates to a large file batch processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a to-be-processed source file and performing virtual fragmentation processing on the source file to obtain a plurality of fragmentation results; sending the plurality of fragmentation results to a plurality of processing nodes; for any processing node, reading data in the fragmentation result through a preset reader component to obtain a plurality of byte arrays; performing service logic processing on the byte array through a preset processor component to obtain a processing result; writing the processing result into a temporary data file through a preset writer component; the processing results in the temporary data file are written into a database, and whether all the processing results are successfully written into the database or not is judged; and if all the processing results are successfully written into the database, deleting the temporary data file. By adopting the method, the data importing efficiency of the large file can be improved, and the consistency of data processing and the high availability of the system are guaranteed.
Need to check novelty before this filing date? Find Prior Art