Large file processing method and device and electronic equipment

Abstract

The invention discloses a large file processing method and device and electronic equipment, and the method comprises the steps: obtaining meta information of a to-be-transmitted file corresponding to a file transmission request in response to the file transmission request; based on the meta information and a preset fragment size, fragment information of the to-be-transmitted file is determined, and the fragment information comprises the fragment size of each fragment; based on the fragment information, utilizing multiple threads to respectively read fragment data of the file to be transmitted, and compressing the fragment data to obtain a plurality of compressed data blocks; and writing the plurality of compressed data blocks into a preset file, and reading each compressed data block from the preset file by using multiple threads so as to transmit each compressed data block. Through the method, the large file processing energy efficiency and timeliness can be improved, and system overload and memory overflow can be avoided. In addition, the transmission stability can be improved, and the method can adapt to complex networks and file transmission scenes.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method, apparatus and electronic device for processing large files. Background Technology

[0002] With the rapid development of cloud computing and big data technologies and the deepening of enterprise informatization, the scale of structured data accumulated in various business systems is growing exponentially. In daily operations, it is frequently necessary to export large amounts of records from databases as large files for uploading, downloading, and sharing to achieve archiving, auditing, exchange, or analysis, such as exporting financial transaction records, packaging user behavior logs, and generating enterprise reports. These operations are crucial for ensuring business continuity and promoting efficient data flow and collaboration. However, the complexity and challenges of large file interactions are becoming increasingly prominent, posing serious challenges to system performance, resource consumption, and business continuity.

[0003] Currently, the industry commonly uses large file processing methods based on general file transfer mechanisms. A typical process involves loading database query results into memory or temporarily storing them as an intermediate file, then splitting and compressing that file, and finally transmitting it via HTTP or other protocols. While this method supports large file transfer to some extent, it still suffers from the following problems: insufficient handling of splitting and merging large files, leading to unstable transmission, chaotic data processing logic, and low efficiency; it requires significant resources, potentially exhausting system resources and impacting other business processes; and it is difficult to adapt to various file transfer scenarios.

[0004] There is currently no effective solution to the problems of resource conflicts, low stability and efficiency, and difficulty in adapting to various file interaction scenarios that are easily caused by file transfer. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, and electronic device for processing large files, in order to solve the problems that file transmission is prone to resource conflicts, has low stability and efficiency, and is difficult to adapt to various file interaction scenarios.

[0006] To solve the above-mentioned technical problems, the first aspect of this specification provides a method for processing large files, including:

[0007] In response to a file transfer request, obtain the metadata of the file to be transferred corresponding to the file transfer request;

[0008] Based on the metadata and the preset fragment size, the fragment information of the file to be transmitted is determined, and the fragment information includes the fragment size of each fragment;

[0009] Based on the fragment information, multiple threads are used to read the fragment data of the file to be transmitted, and each fragment data is compressed to obtain multiple compressed data blocks.

[0010] The multiple compressed data blocks are written to a preset file, and each compressed data block is read from the preset file using multiple threads to transmit each compressed data block.

[0011] In some embodiments of this specification, the metadata includes field data types and estimated values ​​of single-row data size;

[0012] Based on the metadata and the preset fragment size, the fragment information of the file to be transmitted is determined, including:

[0013] Based on the preset fragment size and the estimated value, determine the number of fragment data strips contained in each fragment;

[0014] The number of fragments is determined based on the total number of data segments and the number of fragment data segments of the file to be transmitted;

[0015] The fragment information is determined based on the number of fragments and the number of data entries in each fragment.

[0016] In some embodiments of this specification, the fragmentation information is determined based on the number of fragments and the number of data entries in each fragment, including:

[0017] Based on the number of data entries in each shard and the number of shards, each shard is mapped to a shard query condition. The shard query condition includes an offset and a limit on the number of data entries. The offset is used to represent the number of data entries that need to be skipped when querying the current shard, and the limit on the number of data entries is used to represent the number of shard data entries in the current shard.

[0018] The sharding query conditions are used as the sharding information.

[0019] In some embodiments of this specification, the fragmentation information is determined based on the number of fragments and the number of data entries in each fragment, including:

[0020] Based on the number of data entries in each shard and the number of shards, each shard is mapped to a shard query condition. The shard query condition includes a query offset start point and a query offset end point. The query offset end point includes a query start point field and a query offset end point field.

[0021] The sharding query conditions are used as the sharding information.

[0022] In some embodiments of this specification, based on the fragmentation information, multiple threads are used to read each fragment of the file to be transmitted, and each fragment is compressed to obtain multiple compressed data blocks, including:

[0023] Determine the data type contained in the current data shard;

[0024] Based on the data type, at least one compression algorithm that matches the data type is determined from a plurality of preset compression algorithms;

[0025] Read the current data segment by column or by row, and store the reading results in a preset buffer;

[0026] When the preset buffer reaches a preset storage threshold or the current fragment data is read, the data in the preset buffer is compressed using at least one compression algorithm to obtain the compressed data block corresponding to the current fragment data; and during the data compression process, the data of the next column or the next row is read and written to the preset buffer.

[0027] In some embodiments of this specification, after compressing the data in the preset buffer using the at least one compression algorithm, the method further includes:

[0028] Determine the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block;

[0029] Based on the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block, compressed metadata is generated and written into the compressed data block.

[0030] In some embodiments of this specification, multi-threading is used to read each compressed data block from the preset file separately for transmission of each compressed data block, including:

[0031] Obtain real-time resource metrics data from business systems;

[0032] Based on the real-time resource metrics data, the number of threads or execution priority of multi-threaded systems are dynamically adjusted.

[0033] The dynamic adjustment of multithreading is used to read and transmit each compressed data block.

[0034] In some embodiments of this specification, after writing the plurality of compressed data blocks into a preset file, the method further includes:

[0035] The integrity of the multiple compressed data blocks is verified by comparing the checksum contained in the compressed metadata of each compressed data block with the checksum of the multiple compressed data blocks.

[0036] In some embodiments of this specification, the method further includes:

[0037] Receive multiple compressed fragment files and the manifest file corresponding to the multiple compressed fragment files uploaded by the front end;

[0038] After verifying the integrity of each compressed file segment, the corresponding decompression algorithm is used to perform streaming decompression based on the compressed metadata of each compressed file segment, and the decompressed data is then streamed into the target database.

[0039] In some embodiments of this specification, the decompressed data is streamed into the target database, including:

[0040] The decompressed data is streamed into a preset buffer. Once the preset buffer reaches a preset threshold or the compressed file is written, the data in the preset buffer is streamed into the target database.

[0041] While streaming data to the target database, streaming decompression is performed based on the next data entry or the next compressed file segment.

[0042] The second aspect of this specification provides a large file processing apparatus, comprising:

[0043] The acquisition module is used to acquire the metadata of the file to be transferred corresponding to the file transfer request in response to the file transfer request;

[0044] The determining module is used to determine the fragment information of the file to be transmitted based on the metadata and the preset fragment size, wherein the fragment information includes the fragment size of each fragment;

[0045] The compression module is used to read the data of each segment of the file to be transmitted using multiple threads based on the segment information, and to compress each segment of the data to obtain multiple compressed data blocks.

[0046] The transmission module is used to write the plurality of compressed data blocks into a preset file and use multiple threads to read each compressed data block from the preset file respectively to transmit each compressed data block.

[0047] A third aspect of this specification provides an electronic device, comprising:

[0048] The method comprises a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the steps of the method described in the first aspect above.

[0049] A fourth aspect of this specification provides a computer storage medium storing computer program instructions that, when executed, implement the steps of the method described in the first aspect.

[0050] The fifth aspect of this specification provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0051] The large file processing method, apparatus, and electronic device described in this specification can, in response to a file transfer request, obtain the metadata of the file to be transferred corresponding to the file transfer request; determine the fragment information of the file to be transferred based on the metadata and a preset fragment size, the fragment information including the fragment size of each fragment; based on the fragment information, use multiple threads to read the fragment data of each fragment of the file to be transferred, and compress each fragment data to obtain multiple compressed data blocks; write the multiple compressed data blocks into a preset file, and use multiple threads to read each compressed data block from the preset file to perform the transfer of each compressed data block. By fully utilizing the metadata of the file to be transferred for file splitting, and by using multiple threads to execute the reading, compression, and transfer of fragment data in parallel based on the fragment information, the energy efficiency and timeliness of large file processing can be improved. Furthermore, by using the metadata and the preset fragment size, the fragmentation of the file to be transferred can be precisely controlled, and by combining multi-threaded data reading, compression, and transfer, system overload and memory overflow can be avoided. Furthermore, by segmenting, compressing, and transmitting files, file transmission failures caused by connection timeouts or excessive file size can be avoided, thus improving transmission stability and adapting to complex network and file transmission scenarios. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 The diagram shown is a schematic representation of a large file processing method provided in an embodiment of this specification.

[0054] Figure 2 The diagram shown is a schematic representation of a method for determining fragmentation information provided in an embodiment of this specification.

[0055] Figure 3 The diagram shown is a schematic representation of a compressed data block generation method provided in an embodiment of this specification.

[0056] Figure 4 The diagram shown is a schematic representation of a file import method provided in an embodiment of this specification.

[0057] Figure 5The diagram shown is a schematic of a large file processing device provided in an embodiment of this specification;

[0058] Figure 6 The diagram shown is a schematic of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0060] It should be noted that the information and data related to users involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by the relevant parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with relevant laws, regulations, and standards, and necessary confidentiality measures have been taken. They do not violate public order and good morals, and corresponding operation entry points are provided for users or relevant parties to choose to authorize or refuse.

[0061] It should also be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0062] Before describing the embodiments of this specification, the terms used in the embodiments will be explained.

[0063] MD5: A cryptographic hash function that generates a 128-bit hash value for verifying data integrity.

[0064] Resume interrupted download: If a network transmission fails due to network interruption or other reasons, the transmission can continue from where it left off without having to start from the beginning.

[0065] Buffer: A temporary storage area in memory used to temporarily store input and output data in order to alleviate the speed mismatch between memory and peripherals.

[0066] Concurrent processing: refers to processing multiple tasks simultaneously within the same time period to improve the system's processing capacity and efficiency.

[0067] During large file transfers, excessive bandwidth consumption and strained computing resource allocation often lead to network congestion and system performance degradation. Traditional file processing methods struggle to address performance bottlenecks, high memory usage, and unstable network transmission. Currently, large file transfer primarily utilizes data compression and decompression technologies, as well as asynchronous task processing techniques.

[0068] Data compression and decompression technologies accelerate transmission and effectively alleviate bandwidth pressure by reducing file size. However, their compression efficiency may significantly decrease when processing extremely large files, while system resource consumption remains high. Furthermore, this technology primarily focuses on processing single files, making it difficult to adapt to the complex needs of splitting and merging large files, thus limiting its effectiveness in handling ultra-large-scale data transmission. This method suffers from the following problems: single-file processing limitations—its primary focus on individual files and insufficient handling of splitting and merging of extremely large files—leading to unstable transmission and low efficiency; and, when processing large files, directly loading them into memory results in high memory consumption, especially during multi-user concurrent operations, which can easily exhaust system resources.

[0069] Asynchronous task processing technology, by employing a non-blocking approach, delegates large file transfer tasks to background services, effectively reducing the waiting burden on the front-end or client. However, this technology still has significant shortcomings in areas such as file splitting, resuming interrupted transfers, and error recovery. For example, in the event of network interruptions or system crashes, the transfer process often cannot be automatically resumed, requiring a restart from the beginning, greatly reducing transfer efficiency. Simultaneously, the security risks faced by files during transfer and storage cannot be ignored, as there is a lack of effective data verification mechanisms to ensure data integrity and security. However, this method suffers from the following problems: when transfer is interrupted, it cannot be effectively recovered, requiring retransmission, impacting efficiency and resulting in insufficient transfer reliability; during the transfer and storage of large files, the lack of effective verification mechanisms makes data susceptible to corruption or tampering, affecting data integrity and correctness, and leading to a lack of data security guarantees.

[0070] As can be seen, existing large file transfer methods are prone to resource conflicts, have low stability and efficiency, and are difficult to adapt to various file interaction scenarios. To address these issues, this specification provides a large file processing method that responds to file transfer requests, acquires and analyzes the metadata of the structured data to be transferred, and generates fragmentation information based on the metadata and a preset fragmentation size. This information logically fragments the data and maps it to database query conditions. Furthermore, multiple threads can be started based on the fragmentation information to concurrently perform data extraction, compression, and writing. Through intelligent fragmentation, adaptive compression, and streaming processing, resource conflicts during file transfer are reduced, transmission stability and efficiency are improved, and the method can adapt to complex network and file transfer scenarios.

[0071] It is understood that the methods described in the embodiments of this specification can be applied to electronic devices, which can refer to electronic devices with data computing, processing, and storage capabilities. These electronic devices can be terminals such as PCs (Personal Computers), tablets, smartphones, wearable devices, and intelligent robots; they can also be servers. A server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0072] The large file processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0073] Figure 1 The diagram illustrates a large file processing method provided in an embodiment of this specification. While this specification provides method operation steps or apparatus structures as shown in the following embodiments or figures, the method or apparatus may include more or fewer operation steps or module units, either combined or integrated, based on conventional methods or without inventive effort. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the apparatus is not limited to the execution order or module structure shown in the embodiments or figures of this specification. When the method or module structure is applied in actual devices, servers, or terminal products, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or figures (e.g., in a parallel processor or multi-threaded processing environment, or even in a distributed processing or server cluster implementation environment). Figure 1 As shown, the method may include steps S101 to S104, specifically:

[0074] S101: In response to a file transfer request, obtain the metadata of the file to be transferred corresponding to the file transfer request.

[0075] It is understandable that file transfer requests can be instructions triggered by front-end users (e.g., users clicking "Export Data" or "Batch Upload" buttons) or system scheduled tasks (e.g., automatically archiving data every morning). These requests can include information such as transfer type (import / export), target data range (e.g., "Financial Transaction Records for Month XXXX"), and transfer priority. The metadata of the files to be transferred can be data characterizing the core features of the structured data to be transferred, including but not limited to field data types (e.g., integer, string, date), estimated size of a single row, total number of data rows, source table name, field constraint rules (e.g., NOT NULL, unique), index information (e.g., primary key, partition key, clustered index), and storage characteristics (e.g., whether it is a partitioned table, whether it contains large object fields).

[0076] In practice, when a user initiates a data export or data upload request through the front-end interface, the back-end system receives the request, which typically includes data filtering conditions (such as time range, status, etc.). The system can parse the request and determine the target data source (e.g., the transaction_log table in database db1). Then, the system can connect to this database and execute a lightweight query to obtain metadata about the corresponding data table for subsequent file sharding. The obtained metadata can include field data types, estimated single-row data size, and the total number of data rows. For example, field data types could be BIGINT for id, DECIMAL(10,2) for amount, VARCHAR(255) for comment, and DATETIME for create_time. The estimated single-row data size can be obtained in two ways: (a) querying the database's statistics table to obtain the average row length; (b) performing a weighted estimate based on the standard storage size of the field type. For example, BIGINT occupies 8 bytes, DECIMAL(10,2) is estimated to be 8 bytes, the average length of VARCHAR(255) can be estimated to be 50 bytes based on historical data, and DATETIME occupies 8 bytes. Therefore, the estimated value for a single row is approximately 8 + 8 + 50 + 8 = 74 bytes. The total number of data rows can be obtained by combining the filtering conditions in the request to count the number of data rows. If the data volume is extremely large, the approximate statistical value from the database can be used.

[0077] S102: Based on the metadata and the preset fragment size, determine the fragment information of the file to be transmitted, wherein the fragment information includes the fragment size of each fragment.

[0078] It is understandable that the preset shard size can be a threshold for the maximum data volume of a single shard based on system hardware resources (such as memory capacity and disk I / O speed) and business requirements, such as 200MB. The specific size can be dynamically configured according to the actual scenario (e.g., 100MB in a low-bandwidth environment, 500MB for a high-performance server). Sharding information can include key information for locating and processing shard data, such as the shard size, number of shards, shard query conditions, and data range.

[0079] S103: Based on the fragment information, use multiple threads to read the fragment data of the file to be transmitted, and compress each fragment data to obtain multiple compressed data blocks.

[0080] It is understandable that a compressed data block can be a data unit obtained by each thread processing the fragmented data using a compression algorithm adapted to its data type. It can contain the fragmented data body and compressed metadata (such as compression algorithm identifier, checksum, etc.).

[0081] In practice, when using multiple threads to read data from each shard separately, thread tasks can be assigned to the multiple threads based on the sharding information of each shard.

[0082] S104: Write the plurality of compressed data blocks into a preset file, and use multiple threads to read each compressed data block from the preset file respectively, so as to transmit each compressed data block.

[0083] It is understandable that the preset file can be a temporary folder stored in the temporary storage directory of the backend server, used to aggregate various compressed data blocks, and can support random read / write and multi-threaded concurrent access. The transmission of compressed data blocks can include exporting or uploading compressed data blocks.

[0084] In practice, each thread can write the generated compressed data blocks to a preset file. The file header can store global information such as the number of fragments, the total number of data entries, and the total MD5 value. Each compressed data block is stored sequentially and marked with a fragment number. When transmitting compressed data blocks, each thread pool can read each compressed data block from the preset file and transmit the compressed data blocks to the front end or the target storage node according to the transmission protocol requested by the front end (such as HTTP / HTTPS, FTP). In the case of multi-fragment scenarios, batch download or parallel transmission can be supported. After receiving the data, the front end reassembles the file according to the fragment number.

[0085] In the embodiments described in this specification, file splitting can be fully utilized using the metadata of the file to be transmitted. The reading, compression, and transmission of the split data are then performed in parallel using multiple threads based on the split information, improving the energy efficiency and timeliness of large file processing. Furthermore, by using metadata and preset split sizes, the splitting of the file to be transmitted can be precisely controlled. Combined with multi-threaded data reading, compression, and transmission, system overload and memory overflow can be avoided. In addition, by splitting, compressing, and transmitting files, file transmission failures due to connection timeouts or excessive file size can be avoided, improving transmission stability and adapting to complex network and file transmission scenarios.

[0086] refer to Figure 2 As shown, in some embodiments of this specification, the metadata includes field data types and estimated values ​​of single-row data size. Accordingly, determining the fragment information of the file to be transmitted based on the metadata and a preset fragment size may include:

[0087] S201: Based on the preset fragment size and the estimated value, determine the number of fragment data strips contained in each fragment.

[0088] S202: Determine the number of fragments based on the total number of data segments and the number of fragment data segments of the file to be transmitted.

[0089] S203: Determine the fragment information based on the number of fragments and the number of data entries in each fragment.

[0090] It is understandable that field data types can refer to the storage type of each field in structured data. They can include basic data types (such as INT, VARCHAR, DECIMAL, DATETIME, TINYINT, etc.) and complex data types (such as JSON, ARRAY, etc.). Different field data types can directly affect the space occupied by data storage and the choice of compression algorithm.

[0091] It is understandable that the estimated size of a single row of data can be calculated, for example, by sampling statistics or by accumulating the byte length of field types to obtain the average number of bytes of a single data record. This estimated value can serve as the core basis for partitioning to ensure that the partition size is accurate and controllable.

[0092] It is understandable that the number of data stripes in a shard can be the number of structured data records contained in a single shard, which can be determined by the preset shard size and the estimated size of a single row of data, to ensure that the size of a single shard does not exceed the preset threshold.

[0093] In practice, metadata may include the following:

[0094] Field data types: Report ID (BIGINT), Department ID (INT), Expense Type (VARCHAR (20)), Amount (DECIMAL (12,2)), Reimbursement Date (DATE), Remarks (VARCHAR (100));

[0095] Estimated size of a single row of data: Based on a sample of 2000 data entries, BIGINT (8 bytes) + INT (4 bytes) + VARCHAR (20) (average 12 bytes) + DECIMAL (12,2) (10 bytes) + DATE (4 bytes) + VARCHAR (100) (average 30 bytes) = 68 bytes / row;

[0096] The default fragment size is configured to be 300MB (300×1024×1024=314,572,800 bytes).

[0097] Number of data records per shard = Preset shard size ÷ Estimated size of single row = 314,572,800 ÷ 68 ≈ 4,626,070 records / shard, that is, each shard can contain a maximum of 4,626,070 data records.

[0098] Furthermore, assuming the total number of data entries in the file to be transferred, obtained through a database query, is 18,500,000, the number of fragments is approximately 4. The first 3 fragments each contain 4,626,070 data entries, and the 4th fragment contains 18,500,000 - 3 × 4,626,070 = 4,621,790 data entries, ensuring that the size of each fragment does not exceed 300MB. Based on this number of fragments and the number of data entries contained in each fragment, the fragmentation information can be determined.

[0099] In the embodiments of this specification, fragmentation is calculated based on the metadata of structured data, avoiding the memory overflow caused by excessively large fragments or the frequent transmission of excessively small fragments caused by traditional fixed-size fragmentation. This allows for precise control of fragment size. The calculation of the number of data records in the fragment ensures that a single data record will not be split into multiple fragments, solving the defect of traditional fragmentation that destroys the logical boundaries of data. Subsequent reassembly does not require additional processing of record cutting issues, thus ensuring data integrity. Furthermore, it can support dynamic adjustment of the number of fragments according to the preset fragment size, adapting to different hardware resources and network environments. For example, the preset fragment size can be reduced on low-configuration servers to reduce the single-thread processing pressure, providing greater flexibility.

[0100] In some embodiments of this specification, determining the shard information based on the number of shards and the number of data entries in each shard includes: mapping each shard to a shard query condition based on the number of data entries in each shard and the number of shards. The shard query condition includes an offset and a limit on the number of data entries. The offset is used to represent the number of data entries that need to be skipped when querying the current shard, and the limit on the number of data entries is used to represent the number of shard data entries in the current shard. The shard query condition is then used as the shard information.

[0101] As can be understood, the offset can be the number of preceding data rows skipped during a database query. It can be used to locate the starting position of the current shard. For example, when the offset is 4,626,070, the shard data query can start reading from the 4,626,071st data row.

[0102] It is understandable that limiting the number of data rows (Limit) can be the maximum number of data rows returned by the database query, that is, the number of data rows contained in the current shard, which can ensure that the query results do not exceed the shard range.

[0103] It is understandable that sharded query conditions can be database query statement fragments constructed based on offsets and limits on the number of data rows. This can accurately locate the start and end points of data reading, so as to accurately read the data of the corresponding shards and avoid the waste of resources caused by full table scans.

[0104] In the embodiments described in this specification, the sharded query conditions based on offsets and limits on the number of data rows can utilize database indexes (such as the primary key index of report_id) to avoid full table scans, resulting in high query efficiency. It is compatible with the query syntax of mainstream relational databases (such as MySQL, Oracle, and PostgreSQL), requiring no additional database adaptation plugins, simplifying implementation and reducing system integration costs. Furthermore, each thread executes the query independently, isolating data ranges through offsets and limits on the number of data rows, thus avoiding data duplication or omissions caused by multi-threaded queries, ensuring the uniqueness and integrity of sharded data, and providing high security.

[0105] In some embodiments of this specification, determining the shard information based on the number of shards and the number of data entries in each shard may include: mapping each shard to a shard query condition based on the number of data entries in each shard and the number of shards, wherein the shard query condition includes a query offset start point and a query offset end point, wherein the query offset end point includes a query start point field and a query offset end point field; and using the shard query condition as the shard information.

[0106] The query offset start point can be understood as having a start field and a start value. The start field can be an ordered field in the data table (such as a primary key or a time field), and the start value is the shard start value corresponding to that field, which can be used to accurately locate the starting data of a shard. Similarly, the query offset end point can be having an end field and an end value. The end field is the same as the start field, and the end value can be the shard end value corresponding to that field, which can be used to accurately locate the ending data of a shard. An ordered field can be understood as a field with a unique value that supports sorting, such as a primary key ID, an auto-incrementing sequence, or a timestamp, which ensures the consistency of the query results' order and the accuracy of the shard range.

[0107] It is understandable that the query conditions based on offset and limiting the number of data rows in the aforementioned embodiments may have performance degradation issues in large data scenarios (such as offsets exceeding 10 million rows). The query conditions that include the starting point and ending point of the query offset can be more adapted to large data scenarios.

[0108] In practical implementation, taking a data table (assuming the primary key is log_id, an auto-incrementing sequence, and a total of 200 million data rows) as an example, with a preset shard size of 500MB, an estimated single row size of 50 bytes, approximately 10,485,760 data rows per shard, and approximately 20 shards, we can first query the minimum and maximum values ​​of the primary key log_id. Assuming the results are min_id=1 and max_id=200,000,000, based on the query results, we can calculate the log_id range for each shard. For example, shard 1's log_id range is 1-10,485,760, shard 2's is 10,485,761-20,971,520, ..., and shard 20's is 194,174,241-200,000,000. Based on these query results, we can determine the query offset start and end points for each shard and construct shard query conditions accordingly. In the above process, since `log_id` is the primary key, the database can automatically create a B+ tree index for it. BETWEEN conditional queries can directly locate the data range through the index, avoiding the index scan efficiency degradation caused by skipping the first N rows of data when dealing with large datasets due to offset and row count limitations. For example, if the offset of shard 19 is 188,743,680, a query based on offset and row count limitations would require scanning the index up to row 188,743,680 before reading the data. However, a BETWEEN query can directly locate the index position where `log_id` = 188,743,681, significantly improving query efficiency.

[0109] In the embodiments described in this specification, by querying the starting and ending points of the query offset, the performance degradation problem when the offset is too large can be addressed based on the offset and the number of data rows. This approach is suitable for scenarios with large data volumes. It supports multiple ordered fields such as primary keys and time fields, making it compatible with distributed database sharding scenarios and offering stronger compatibility. Furthermore, range queries based on ordered fields can ensure the order of sharded data, avoiding data disorder caused by multi-threaded concurrent queries. Subsequent reorganization does not require additional sorting operations, resulting in high data consistency.

[0110] refer to Figure 3 As shown, in some embodiments of this specification, based on the fragmentation information, multiple threads are used to read the fragment data of the file to be transmitted, and each fragment data is compressed to obtain multiple compressed data blocks, which may include:

[0111] S301: Determine the data type contained in the current fragment data.

[0112] S302: Based on the data type, determine at least one compression algorithm that matches the data type from a plurality of preset compression algorithms.

[0113] S303: Read the current fragment data by column or by row, and store the reading results in a preset buffer.

[0114] S304: When the preset buffer reaches the preset storage threshold or the current fragment data is read, the data in the preset buffer is compressed using the at least one compression algorithm to obtain the compressed data block corresponding to the current fragment data; and during the data compression process, the data of the next column or the next row is read and written to the preset buffer.

[0115] It is understandable that the preset compression algorithm library can be a collection of pre-integrated compression algorithms, which may include general compression algorithms (LZ4, gzip, Snappy) and compression algorithms specifically for structured data (differential coding, dictionary compression, fixed-point compression, RLE coding), and can support dynamic selection based on data type.

[0116] It is understandable that the preset buffer can be a temporary storage area in memory, which can be used to temporarily store the read fragmented data. The buffer size is configurable and can support two triggering mechanisms: full threshold compression and data read completion compression.

[0117] It is understandable that streaming compression can perform compression processing while reading data, that is, reading, buffering, and compression are executed in parallel, without waiting for all the data in the fragment to be read before compression, which can reduce processing latency.

[0118] In practice, the thread queries the database based on the query conditions in the sharding information, obtaining a database cursor or streaming result set. Column-based buffering allocates an independent buffer for each field involved in the current shard. When a record is read from the cursor, each field value of that record is appended to the corresponding field's buffer. Simultaneously, preprocessing is performed based on the field type (e.g., calculating the difference between the integer field and the previous value). Row-based buffering (suitable for row-based processing or small records) caches several complete records in a single buffer. When the buffer for a certain field reaches a preset storage threshold (e.g., 64KB), or when all data in the current shard has been read, compression of the buffer data is triggered. During compression, a corresponding algorithm is selected from the compression strategy mapping table based on the field's data type. The compressed data block is immediately appended to the temporary file corresponding to that shard. While compressing the current buffer data, the thread can continue reading the next record / batch from the database cursor, filling the next buffer, thus implementing a pipelined "read-preprocess-compress-write" operation.

[0119] In some embodiments of this specification, the system may preset a compression strategy mapping table, that is, a mapping relationship between multiple preset compression algorithms and data types. For example, the preset compression strategy mapping table may be:

[0120] Data types INTEGER, BIGINT, and DATETIME can be compressed using the algorithm DELTA_ENCODING (differential encoding) + VARINT (variable-length integer encoding). Data types VARCHAR and TEXT with enumerable value ranges (e.g., the status field has values ​​of "SUCCESS" or "FAILED") can be compressed using the algorithm DICTIONARY_ENCODING (dictionary encoding) + HUFFMAN (Huffman encoding). Data types VARCHAR and TEXT with non-enumerable value ranges can be compressed using the algorithm ZSTD (streaming compression). Data type DECIMAL can be compressed using the algorithm DELTA_ENCODING after converting to an integer (multiplying by a fixed coefficient). Data type BLOB can be compressed using the algorithm LZ4 (high-speed lossless compression).

[0121] It is understood that the above-mentioned preset compression strategy mapping table is only one example of this specification. Other embodiments may include more or fewer data types and compression algorithms, or other mapping relationships may be used between data types and compression algorithms. This specification does not limit this.

[0122] In the embodiments described in this specification, targeted column-aware compression achieves a higher compression ratio (saving storage and transmission space) compared to general compression algorithms (such as gzip), and also offers faster compression / decompression speeds because the algorithm is more adapted to data characteristics. By combining streaming processing with a buffer mechanism, it can be ensured that only a small portion of the data resides in memory at any given time, avoiding the risk of Out of Memory (OOM) errors caused by loading the entire fragment or even the entire dataset into memory. Furthermore, the pipelined job mode allows data reading, compression, and writing to partially overlap, reducing overall processing time.

[0123] In some embodiments of this specification, after compressing the data in the preset buffer using the at least one compression algorithm, the process may further include: determining the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block; generating compressed metadata based on the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block; and writing the compressed metadata into the compressed data block.

[0124] It is understood that for each compressed data block, a piece of compressed metadata is generated and written to its header or footer when writing to a temporary file. For example, this metadata may include:

[0125] Compression algorithm identifier: such as ZSTD, DELTA_VARINT.

[0126] Original data size: The size of this block of data before compression.

[0127] Size of compressed data: The size of this compressed data block.

[0128] Checksum: The CRC32 or MD5 fragment value calculated for this compressed data block.

[0129] Data bar information: This compressed data block originates from record number X to record Y of the original fragment.

[0130] Dictionary information (if dictionary compression is used): Dictionary version or the dictionary content itself (if the dictionary is small).

[0131] In addition, after the entire sharding process is completed, a shard-level metadata header can be generated, recording the total number of records in the shard, the total list of compression algorithms used, the shard-level checksum, etc., and written to the beginning of a temporary file.

[0132] In this embodiment, compression metadata records key information such as the compression algorithm and data range. This allows for precise selection of the corresponding algorithm during subsequent decompression, preventing decompression failures. It also supports data bar-level positioning and verification, ensuring strong data traceability. Furthermore, verification values ​​provide dual protection for data integrity. Combined with the overall file MD5 checksum, it effectively identifies data tampering or transmission corruption, guaranteeing data integrity. In addition, the reserved metadata extension area supports the addition of new attributes (such as data encryption identifiers and transmission priorities) without modifying the core storage format, demonstrating strong scalability.

[0133] In some embodiments of this specification, using multiple threads to read each compressed data block from the preset file for transmission may include: acquiring real-time resource indicator data of the business system; dynamically adjusting the number of threads or execution priority of the multiple threads based on the real-time resource indicator data; and using the dynamically adjusted multiple threads to read and transmit each compressed data block.

[0134] Real-time resource metrics can be understood as indicators of the hardware and software resource usage of backend servers, including CPU utilization, memory usage, disk I / O speed, network bandwidth utilization, and database connection count. The sampling period for these real-time resource metrics can be preset, for example, to 1 second. Dynamic adjustment of the number of threads can be based on real-time resource metrics, increasing or decreasing the number of threads used for data reading, compression, and transmission to ensure resource usage does not exceed preset thresholds. Execution priority adjustment can involve assigning different operating system priorities to threads. When system resources are strained, the priority of file transfer threads can be reduced to ensure resource utilization for online business threads.

[0135] In the embodiments described in this specification, by dynamically adjusting the number and priority of threads, it is possible to ensure that the large file transfer service does not excessively consume resources, guarantee the stability and response rate of CPU usage for online services, achieve "smooth resource consumption," and avoid resource conflicts. When system resources are sufficient, the number of threads can be maximized to improve processing efficiency; when resources are scarce, the number of threads can be reasonably reduced to ensure core business operations, achieving optimal resource utilization. Furthermore, it can adapt to different hardware configurations and business load scenarios without manual intervention in thread pool configuration, reducing operational costs and demonstrating strong adaptability.

[0136] In some embodiments of this specification, after writing the plurality of compressed data blocks into a preset file, the method may further include: comparing the checksum contained in the compressed metadata of each compressed data block with the checksum of the plurality of compressed data blocks to verify the integrity of the plurality of compressed data blocks.

[0137] It's understandable that the fragment checksum can be the MD5 checksum of each compressed data block, stored in the compressed metadata, and used to verify the integrity of a single fragment. The total checksum of multiple compressed data blocks can be the MD5 checksum calculated by concatenating all compressed data blocks according to their fragment numbers, stored in the header of a pre-defined file, and used to verify the integrity of the entire file. You can first verify the fragment checksum of a single compressed data block, and then verify the total checksum after concatenating all fragments, ensuring that the data is undamaged, untampered with, and complete.

[0138] In practice, the MD5 value of each compressed data block can be calculated when generating each compressed data block. After the compressed data blocks are written to a preset file, the total MD5 value of all compressed data blocks can be calculated. Then, during data verification, the backend server can recalculate the MD5 value of each compressed data block before transmitting it and compare it with the fragment checksum in the compressed metadata. For example, if the recalculated MD5 value of fragment 1 is A1B2C3D4E5F6A7B8C9D0E1F2A3B4C5D6, it matches the metadata, and the verification passes. If the checksum of a fragment is inconsistent (e.g., the recalculated result of fragment 2 is A2B3C4D5E6F7A8B9C0D1E2F3A4B5C6D8), then the fragment is considered corrupted, triggering the process of regenerating the fragment. After all fragments have been transmitted, the front end receives and concatenates all compressed data blocks according to their sequence numbers. It can calculate the MD5 value of the complete data stream and compare it with the total checksum sent by the back end. If they match, the integrity of the entire file is determined, and subsequent decompression and reassembly can proceed. If they do not match, the front end prompts that the file transmission is incomplete and requests a re-download. It also returns the sequence number of the corrupted fragment. The back end only needs to retransmit that fragment and does not need to retransmit the entire file.

[0139] In the embodiments described in this specification, the above-mentioned dual verification mechanism can identify tampering, damage and omissions during data transmission, solving the data error problem caused by the lack of effective verification in traditional solutions, and ensuring high data integrity. When a single fragment verification fails, only that fragment needs to be retransmitted, without the need for a full retransmission, saving bandwidth resources, which is especially suitable for the transmission of very large files. The MD5 checksum is irreversible, which can effectively resist data tampering attacks and ensure the transmission security of structured data (such as financial data and transaction records).

[0140] refer to Figure 4 As shown, in some embodiments of this specification, the above method may further include:

[0141] S401: Receives multiple compressed fragment files and the manifest file corresponding to the multiple compressed fragment files uploaded by the front end.

[0142] S402: After verifying the integrity of each compressed file segment, perform streaming decompression using the corresponding decompression algorithm based on the compressed metadata of each compressed file segment, and stream the decompressed data to the target database.

[0143] It is understandable that the manifest file can contain core information about all compressed fragment files, such as the number of fragments, total checksum, filename of each fragment, fragment sequence number, fragment checksum, compression algorithm, etc., for fragment verification and reassembly during import. Streaming decompression allows decompression processing to be performed simultaneously with receiving compressed fragment files, without waiting for all fragments to be received, reducing storage usage and processing latency. The target database can be the database receiving the imported data, which can be a relational database, data warehouse, or distributed file system, and can support streaming data writing. It is understood that step S401 above can be a file upload transmission scenario, which is the opposite of the aforementioned file export process.

[0144] In the embodiments described in this specification, parallel uploading, streaming decompression, and pipelined writing achieve higher import speed and efficiency compared to the traditional full-reception-full-decompression-full-write scheme. It can support fragment-level error recovery, eliminating the need for full retransmission if a single fragment fails to transmit or decompress, thus reducing the risk of import failure. Streaming processing avoids storing the full compressed file or decompressed data on disk, making it suitable for large batches of data.

[0145] In some embodiments of this specification, streaming the decompressed data into the target database may include: streaming the decompressed data into a preset buffer; streaming the data in the preset buffer into the target database when the preset buffer reaches a preset threshold or when the compressed fragment file is written; and streaming decompression based on the next data or the next compressed fragment file while streaming the data into the target database.

[0146] This specification also provides a method for large file interaction in the embodiments, which may include large file export and large file import.

[0147] Specifically, exporting large files can include the following stages:

[0148] 1) File query and writing

[0149] The backend system first splits the large file into chunks based on the user request, determines the chunk information, and then queries the data to be exported based on the chunk information. The query results are then written to a temporary folder, with each file containing no more than a preset threshold (using a threshold of 200MB as an example). Data exceeding the threshold is automatically split and corresponding file packages are generated.

[0150] 2) Multi-threaded writing and compression

[0151] The system employs multi-threading to simultaneously handle the writing and compression of multiple file packets (i.e., file fragments). Each thread is responsible for writing and compressing one or more file packets, while controlling memory usage to not exceed a preset limit (e.g., 3GB). Appropriate compression algorithms are configured for different data types to compress the file packets, reducing storage space usage and transmission time.

[0152] 3) Naming rules and MD5 verification

[0153] A unique naming convention is generated for each file package, including information such as filename, file sequence number, total number of files, and total MD5 value of the files. After the file package is written, its MD5 value is calculated and compared with the total MD5 value of the files to verify file integrity.

[0154] 4) Notification download and resume interrupted download

[0155] Once all data has been written and compressed, the backend system notifies the frontend user that they can begin downloading files. The frontend provides a unified download link to support batch downloading of multiple file packages and supports resume functionality to allow downloads to resume if the network is interrupted.

[0156] 5) Cleanup mechanism

[0157] After data export is complete, set up a scheduled task to clean up file packages in the temporary folder (default is 1 hour) to free up disk space. Intermediate files generated during processing are cleaned up immediately to avoid excessive resource consumption.

[0158] Specifically, importing large files can include the following stages:

[0159] 1) Single-file import mechanism

[0160] Users upload compressed files of 200MB or less via the front-end interface. After upload, the back-end system verifies file integrity and writes the files to a specified location on the disk. The data on the disk is then imported into the database, and the file status is updated to indicate upload success or failure. After import, files on the disk are cleaned up to free up space.

[0161] 2) Multi-file import mechanism

[0162] Users can select multiple compressed file packages to upload at once. The backend system processes each uploaded file package one by one and updates its status to either upload success or failure. After all file packages have been uploaded, it checks whether the total number of files in the filenames matches the actual number of files uploaded; if they do not match, it prompts the user that there is a risk of packet loss and requires them to reconfirm whether the uploaded files are complete. If the total number of files matches the number of uploaded files, it performs a file merging operation and calculates the MD5 value of the merged file, comparing it with the MD5 value of the uploaded file to verify file integrity. Using multi-threading and buffer management techniques, the merged file data is written to the database in batches, and the file status is updated to either import success or failure. After the import is complete, the frontend user is notified, and files on the disk are cleaned up to free up space.

[0163] In other embodiments, the processes of splitting, compressing, and transmitting large files can also be implemented using cloud storage and computing services. That is, large file processing can be achieved using the distributed storage and computing services provided by a cloud computing platform. This approach fully leverages the resource advantages and elastic scalability of the cloud platform.

[0164] In the embodiments of this specification, the following technical effects can be achieved through the above-described large file interaction method:

[0165] High efficiency: The intelligent splitting and merging mechanism and multi-threaded processing technology significantly improve the processing efficiency of large files.

[0166] Stability: The memory management and buffer control mechanisms effectively avoid the risks of memory overflow and system crashes, ensuring stable system operation.

[0167] Security: Mechanisms such as MD5 verification ensure the integrity and security of files during transmission and storage.

[0168] Excellent user experience: Support for resume download function provides a flexible operation interface and friendly prompts, which significantly improves the user experience.

[0169] Maintainability: The modular design makes the system easy to maintain and expand, and it can be easily customized and optimized according to actual needs.

[0170] Based on the large file processing method described above, one or more embodiments of this specification also provide a large file processing apparatus. The apparatus may include devices (including distributed systems), software (applications), modules, plug-ins, servers, clients, etc., using the methods described in the embodiments of this specification, combined with necessary hardware implementations. Based on the same innovative concept, the apparatuses in one or more embodiments provided in this specification are as described in the following embodiments. Since the implementation schemes and methods for solving the problem are similar, the implementation of specific apparatuses in the embodiments of this specification can refer to the implementation of the foregoing methods, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatuses described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated. Figure 5 The diagram shown is a schematic representation of a large file processing device provided in an embodiment of this specification. Figure 5 As shown, the large file processing device 500 may include [the following].

[0171] The acquisition module 501 is used to acquire the metadata of the file to be transferred corresponding to the file transfer request in response to the file transfer request.

[0172] The determining module 502 is used to determine the fragment information of the file to be transmitted based on the metadata and the preset fragment size, wherein the fragment information includes the fragment size of each fragment.

[0173] Compression module 503 is used to read each segment of the file to be transmitted using multiple threads based on the segment information, and to compress each segment of the data to obtain multiple compressed data blocks.

[0174] The transmission module 504 is used to write the plurality of compressed data blocks into a preset file and use multiple threads to read each compressed data block from the preset file respectively to transmit each compressed data block.

[0175] In some embodiments of this specification, the metadata may include field data types and an estimated value of the size of a single row of data; the determining module 502 may be specifically used to: determine the number of data segments contained in each segment based on the preset segment size and the estimated value; determine the number of segments based on the total number of data segments and the number of data segments in the file to be transmitted; and determine the segment information based on the number of segments and the number of data segments in each segment.

[0176] In some embodiments of this specification, when determining the shard information based on the number of shards and the number of data entries in each shard, the determining module 502 may specifically be used to: map each shard to a shard query condition based on the number of data entries in each shard and the number of shards, wherein the shard query condition includes an offset and a limit on the number of data entries, wherein the offset is used to characterize the number of data entries to be skipped when querying the current shard, and the limit on the number of data entries is used to characterize the number of shard data entries in the current shard; and use the shard query condition as the shard information.

[0177] In some embodiments of this specification, when determining the shard information based on the number of shards and the number of data entries in each shard, the determining module 502 may specifically be used to: map each shard to a shard query condition based on the number of data entries in each shard and the number of shards, wherein the shard query condition includes a query offset start point and a query offset end point, wherein the query offset end point includes a query start point field and a query offset end point field; and use the shard query condition as the shard information.

[0178] In some embodiments of this specification, the compression module 503 may specifically be used to: determine the data type contained in the current fragment data; based on the data type, determine at least one compression algorithm that matches the data type from a plurality of preset compression algorithms; read the current fragment data column by column or row by row, and store the reading result in a preset buffer; when the preset buffer reaches a preset storage threshold or the current fragment data is read completely, compress the data in the preset buffer using the at least one compression algorithm to obtain a compressed data block corresponding to the current fragment data; and synchronously read the data of the next column or the next row and write it into the preset buffer during the data compression process.

[0179] In some embodiments of this specification, the compression module 503 may also be used to: determine the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block; generate compressed metadata based on the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block; and write the compressed metadata into the compressed data block.

[0180] In some embodiments of this specification, the transmission module 504 may be specifically used to: acquire real-time resource indicator data of the business system; dynamically adjust the number of threads or execution priority of the multi-threaded system based on the real-time resource indicator data; and use the dynamically adjusted multi-threaded system to read and transmit each compressed data block.

[0181] In some embodiments of this specification, the above-described apparatus may further include a verification module for: comparing the checksum contained in the compressed metadata of each compressed data block with the checksum of the plurality of compressed data blocks to verify the integrity of the plurality of compressed data blocks.

[0182] In some embodiments of this specification, the above-described apparatus may further include an import module, configured to: receive multiple compressed fragment files and manifest files corresponding to the multiple compressed fragment files uploaded by the front end; after verifying the integrity of each compressed fragment file, perform streaming decompression according to the compressed metadata of each compressed fragment file using the corresponding decompression algorithm, and stream the decompressed data into the target database.

[0183] In some embodiments of this specification, when the import module streams the decompressed data into the target database, it can specifically be used to: stream the decompressed data into a preset buffer; when the preset buffer reaches a preset threshold or the compressed fragment file is written, stream the data in the preset buffer into the target database; while streaming data into the target database, stream decompress the data according to the next data or the next compressed fragment file.

[0184] The descriptions and functions of the above modules can be found in the section on large file processing methods, and will not be repeated here.

[0185] This application also provides an electronic device, such as... Figure 6 As shown, the electronic device 600 may include a processor 601 and a memory 602, wherein the processor 601 and the memory 602 may be connected via a bus 603 or other means. Figure 6 Taking the connection via bus 603 as an example.

[0186] In some embodiments of this specification, processor 601 may be a central processing unit (CPU). In other embodiments, processor 601 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0187] The memory 602, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the large file processing method in the embodiments of this application. The processor 601 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory 602, thereby implementing the large file processing method in the above method embodiments.

[0188] The memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor 601, etc. Furthermore, the memory 602 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 602 may optionally include memory remotely located relative to the processor 601, and these remote memories may be connected to the processor 601 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0189] The one or more modules are stored in the memory 602, and when executed by the processor 601, the following large file processing method is performed:

[0190] In response to a file transfer request, the system obtains the metadata of the file to be transferred corresponding to the file transfer request; based on the metadata and a preset fragment size, it determines the fragment information of the file to be transferred, the fragment information including the fragment size of each fragment; based on the fragment information, it uses multiple threads to read the fragment data of each fragment of the file to be transferred, and compresses each fragment data to obtain multiple compressed data blocks; it writes the multiple compressed data blocks into a preset file, and uses multiple threads to read each compressed data block from the preset file to perform the transfer of each compressed data block.

[0191] The specific details of the aforementioned electronic device 600 can be understood by referring to the relevant descriptions and effects in the above method embodiments, and will not be repeated here.

[0192] This specification also provides a computer storage medium storing computer program instructions, which, when executed, implement the steps of the above-described large file processing method.

[0193] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0194] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, please refer to each other. The focus of each embodiment is to describe the differences from other embodiments.

[0195] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0196] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0197] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute certain parts of the methods of various embodiments of this application.

[0198] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0199] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0200] Although this application has been described through embodiments, those skilled in the art will know that this application has many modifications and variations without departing from the spirit of this application, and it is intended that the appended claims cover such modifications and variations without departing from the spirit of this application.

Claims

1. A method for processing large files, characterized in that, include: In response to a file transfer request, obtain the metadata of the file to be transferred corresponding to the file transfer request; Based on the metadata and the preset fragment size, the fragment information of the file to be transmitted is determined, and the fragment information includes the fragment size of each fragment; Based on the fragment information, multiple threads are used to read the fragment data of the file to be transmitted, and each fragment data is compressed to obtain multiple compressed data blocks. The multiple compressed data blocks are written to a preset file, and each compressed data block is read from the preset file using multiple threads to transmit each compressed data block.

2. The large file processing method according to claim 1, characterized in that, The metadata includes field data types and estimated values ​​for the size of a single row of data; Based on the metadata and the preset fragment size, the fragment information of the file to be transmitted is determined, including: Based on the preset fragment size and the estimated value, determine the number of fragment data strips contained in each fragment; The number of fragments is determined based on the total number of data segments and the number of fragment data segments of the file to be transmitted; The fragment information is determined based on the number of fragments and the number of data entries in each fragment.

3. The large file processing method according to claim 2, characterized in that, The fragmentation information is determined based on the number of fragments and the number of data entries in each fragment, including: Based on the number of data entries in each shard and the number of shards, each shard is mapped to a shard query condition. The shard query condition includes an offset and a limit on the number of data entries. The offset is used to represent the number of data entries that need to be skipped when querying the current shard, and the limit on the number of data entries is used to represent the number of shard data entries in the current shard. The sharding query conditions are used as the sharding information.

4. The large file processing method according to claim 3, characterized in that, The fragmentation information is determined based on the number of fragments and the number of data entries in each fragment, including: Based on the number of data entries in each shard and the number of shards, each shard is mapped to a shard query condition. The shard query condition includes a query offset start point and a query offset end point. The query offset end point includes a query start point field and a query offset end point field. The sharding query conditions are used as the sharding information.

5. The large file processing method according to claim 1, characterized in that, Based on the fragmentation information, multiple threads are used to read each fragment of the file to be transmitted, and each fragment is compressed to obtain multiple compressed data blocks, including: Determine the data type contained in the current data shard; Based on the data type, at least one compression algorithm that matches the data type is determined from a plurality of preset compression algorithms; Read the current data segment by column or by row, and store the reading results in a preset buffer; When the preset buffer reaches the preset storage threshold or the current fragment data is read, the data in the preset buffer is compressed using the at least one compression algorithm to obtain the compressed data block corresponding to the current fragment data; and during the data compression process, the data of the next column or the next row is read and written to the preset buffer.

6. The large file processing method according to claim 5, characterized in that, After compressing the data in the preset buffer using the at least one compression algorithm, the method further includes: Determine the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block; Based on the compression algorithm, compressed data size, checksum, data bar information, and fragment size corresponding to the compressed data block, compressed metadata is generated and written into the compressed data block.

7. The large file processing method according to claim 1, characterized in that, The process involves using multiple threads to read each compressed data block from the preset file, and then transmitting each compressed data block. Obtain real-time resource metrics data from business systems; Based on the real-time resource metrics data, the number of threads or execution priority of multi-threaded systems are dynamically adjusted. The dynamic adjustment of multithreading is used to read and transmit each compressed data block.

8. The large file processing method according to claim 1, characterized in that, After writing the plurality of compressed data blocks into the preset file, the process further includes: The integrity of the multiple compressed data blocks is verified by comparing the checksum contained in the compressed metadata of each compressed data block with the checksum of the multiple compressed data blocks.

9. The large file processing method according to claim 1, characterized in that, Also includes: Receive multiple compressed fragment files and the manifest file corresponding to the multiple compressed fragment files uploaded by the front end; After verifying the integrity of each compressed file segment, the corresponding decompression algorithm is used to perform streaming decompression based on the compressed metadata of each compressed file segment, and the decompressed data is then streamed into the target database.

10. The large file processing method according to claim 9, characterized in that, The decompressed data is streamed into the target database, including: The decompressed data is streamed into a preset buffer. Once the preset buffer reaches a preset threshold or the compressed fragment file is written, the data in the preset buffer is streamed into the target database. While streaming data to the target database, streaming decompression is performed based on the next data entry or the next compressed file segment.

11. A large file processing device, characterized in that, include: The acquisition module is used to acquire the metadata of the file to be transferred corresponding to the file transfer request in response to the file transfer request; The determining module is used to determine the fragment information of the file to be transmitted based on the metadata and the preset fragment size, wherein the fragment information includes the fragment size of each fragment; The compression module is used to read the data of each segment of the file to be transmitted using multiple threads based on the segment information, and to compress each segment of the data to obtain multiple compressed data blocks. The transmission module is used to write the plurality of compressed data blocks into a preset file and use multiple threads to read each compressed data block from the preset file respectively to transmit each compressed data block.

12. An electronic device, characterized in that, include: A memory and a processor, the processor and the memory being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to implement the steps of the method according to any one of claims 1 to 10.

13. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed, implement the steps of the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images