File loading method, device, electronic device and storage medium

Abstract

The present application discloses a file loading method, device, electronic device and storage medium. The present application divides the file to be loaded into various main blocks according to a preset first data volume. After determining the number of file loading processes based on the configuration information of the application system resources, the main blocks are allocated to the corresponding processes according to the load balancing strategy. In each process, the loading data corresponding to the main block is determined based on the starting position information of the main block and the preset first data volume, and the loading data is loaded. Because the data volume of each main block of the present application is the same, each main block is allocated according to the load balancing strategy, and each process processes each main block in parallel. In this way, the amount of data loaded by each process is the same, and the processing time is similar, which solves the problem of the existing technology that the total processing time is long due to the large difference in file data volume.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a file loading method, device, electronic device and storage medium. Background Art

[0002] When an application system processes tens of thousands of files, each containing a different number of records, and a single process loads (reads, parses, and stores records) each file, while multiple files can be processed concurrently, processing a single file with a large transaction volume will inevitably lead to significant delays, impacting overall processing time. Therefore, completing the processing of all files in the shortest possible time becomes a pressing issue. Summary of the Invention

[0003] The embodiments of the present application provide a file loading method, device, electronic device and storage medium to solve the problem in the prior art of long total processing time due to large differences in file data volume when loading files.

[0004] In a first aspect, the present application provides a file loading method, the method comprising:

[0005] Dividing the file to be loaded according to a preset first data amount to obtain respective main blocks of the file, and recording starting position information of the respective main blocks in the file;

[0006] Obtain and determine the number of file loading processes based on the configuration information of application system resources;

[0007] According to the load balancing strategy, each main block is allocated to the corresponding process. When the corresponding main block is loaded in the process, the loading data corresponding to the main block is determined according to the starting position information of the main block and the preset first data volume, and the loading data is loaded.

[0008] In a second aspect, the present application provides a file loading device, the device comprising:

[0009] a partitioning module, configured to partition the file to be loaded according to a preset first data volume to obtain respective primary blocks of the file, and record starting position information of the respective primary blocks in the file;

[0010] A determination module is used to obtain and determine the number of file loading processes based on the configuration information of the application system resources;

[0011] The loading module is used to allocate the various main blocks to the corresponding processes according to the load balancing strategy. When loading the corresponding main blocks in the process, the loading data corresponding to the main blocks is determined according to the starting position information of the main blocks and the preset first data volume, and the loading data is loaded.

[0012] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0013] Memory for storing computer programs;

[0014] The processor is used to implement the method steps when executing the program stored in the memory.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described are implemented.

[0016] The present application provides a file loading method, device, electronic device and storage medium, the method comprising: dividing the file to be loaded according to a preset first data volume to obtain each main block of the file, and recording the starting position information of each main block in the file; obtaining and determining the number of file loading processes based on the configuration information of the application system resources; allocating the each main block to the corresponding process according to the load balancing strategy, and when loading the corresponding main block in the process, determining the loading data corresponding to the main block according to the starting position information of the main block and the preset first data volume, and loading the loading data.

[0017] The above technical solution has the following advantages or beneficial effects:

[0018] The present application divides the file to be loaded into various main blocks according to the preset first data volume. After determining the number of file loading processes based on the configuration information of the application system resources, the main blocks are allocated to the corresponding processes according to the load balancing strategy. In each process, the loading data corresponding to the main blocks is determined based on the starting position information of the main blocks and the preset first data volume, and the loading data is loaded. Because the data volume of each main block of the present application is the same, each main block is allocated according to the load balancing strategy, and each process processes each main block in parallel. In this way, the amount of data loaded by each process is the same, and the processing time is similar, which solves the problem of the existing technology that the total processing time is long due to the large difference in file data volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 Schematic diagram of the file loading process provided for this application;

[0021] Figure 2 This is a schematic diagram of a mode in which multiple processes are loaded concurrently and a single file is loaded by a single process in the prior art;

[0022] Figure 3 This is a schematic diagram of a mode in which multiple processes concurrently load a single file in the prior art;

[0023] Figure 4 A document numbering flow chart provided for this application;

[0024] Figure 5 Flowchart of the logical division of the files provided for this application;

[0025] Figure 6 Flowchart of logical block and batch division of files provided for this application;

[0026] Figure 7 Schematic diagram of the logical block grouping process of files provided for this application;

[0027] Figure 8 Schematic diagram of the logical block data reading solution provided by this application;

[0028] Figure 9 The overall flow chart of concurrent loading of files provided for this application;

[0029] Figure 10 A flowchart of asynchronous file loading, reading and writing provided by this application;

[0030] Figure 11 Breakpoint continuation flow chart provided for this application;

[0031] Figure 12 A schematic diagram of the file loading device structure provided for this application;

[0032] Figure 13 This is a schematic diagram of the electronic device structure provided in this application. DETAILED DESCRIPTION

[0033] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0034] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0035] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0036] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0037] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functionality associated with that element.

[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0039] For ease of explanation, the above description has been made with reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments are selected and described to better explain the principles and practical applications, so that those skilled in the art can better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0040] Figure 1 The file loading process provided for this application includes the following steps:

[0041] S101: Divide the file to be loaded according to a preset first data volume to obtain primary blocks of the file, and record starting position information of the primary blocks in the file.

[0042] S102: Obtain and determine the number of file loading processes based on configuration information of application system resources.

[0043] S103: Allocate the respective master blocks to corresponding processes according to the load balancing strategy. When loading the corresponding master blocks in the process, determine the loading data corresponding to the master blocks based on the starting position information of the master blocks and the preset first data volume, and load the loading data.

[0044] The file loading method provided in this application is applied to electronic devices, which may be computers, servers, and other devices.

[0045] The electronic device first obtains the file to be loaded, and then divides the file to be loaded according to a preset first data amount to obtain each main block of the file. The preset first data amount is, for example, 10M, 15M, etc. The data amount of the last main block may be smaller than the preset first data amount. The electronic device divides the file to be loaded according to the preset first data amount in a certain order, so that the starting position information of each main block in the file can be recorded. The starting position information of each main block in the file is known, and each main block is obtained by dividing according to the preset first data amount, so the end position information of each main block in the file can also be determined.

[0046] The electronic device obtains configuration information for application system resources and, based on the configuration information, determines the number of file loading processes. Each master block is then assigned to a corresponding process according to a load balancing strategy. When loading a master block, each process first obtains the master block's starting position information and a preset first data size. Based on the master block's starting position information and the preset first data size, it determines the loading data corresponding to the master block, thereby completing the loading of the data.

[0047] The present application divides the file to be loaded into various main blocks according to the preset first data volume. After determining the number of file loading processes based on the configuration information of the application system resources, the main blocks are allocated to the corresponding processes according to the load balancing strategy. In each process, the loading data corresponding to the main blocks is determined based on the starting position information of the main blocks and the preset first data volume, and the loading data is loaded. Because the data volume of each main block of the present application is the same, each main block is allocated according to the load balancing strategy, and each process processes each main block in parallel. In this way, the amount of data loaded by each process is the same, and the processing time is similar, which solves the problem of the existing technology that the total processing time is long due to the large difference in file data volume.

[0048] Considering that when a file is divided into multiple primary blocks, a transaction may be divided into different primary blocks. In order to ensure the integrity of data loading, the method in this application further includes:

[0049] For each of the primary blocks, determining an additional block corresponding to the primary block, wherein the additional block includes data after the end of the primary block in the file;

[0050] Allocating the respective primary blocks and the corresponding additional blocks to corresponding processes according to a load balancing strategy, and when loading the corresponding primary blocks in the processes, determining first loading data corresponding to the primary blocks according to the starting position information of the primary blocks and the preset first data volume;

[0051] Determine the second loaded data before the first complete record in the additional block corresponding to the main block;

[0052] The second loading data is added to the main block, and the first loading data and the second loading data are loaded.

[0053] This application divides the file to be loaded into primary blocks and, for each primary block, determines the corresponding additional block. The additional block is sized to at least contain the size of a complete record and contains the data after the end of the primary block in the file. When loading the file, each primary block and its corresponding additional block are assigned to the corresponding process according to a load balancing strategy.

[0054] When loading the corresponding primary block during the process, the first loading data corresponding to the primary block is first determined based on the primary block's starting position information and the preset first data volume. Then, the second loading data preceding the first complete record in the additional block corresponding to the primary block is determined. The second loading data is the data that is separated from the same transaction in the primary block. The second loading data is then added to the primary block. This completes the loading data corresponding to the primary block, i.e., the first loading data and the second loading data are loaded. This ensures the integrity of the data loading even when a file is divided into multiple primary blocks and a transaction is split into different primary blocks.

[0055] In this application, for each of the main blocks, determining the additional block corresponding to the main block includes:

[0056] For each of the main blocks, determine the end position information of the main block, take the end position information as the starting point, divide the file according to a preset second data amount, and obtain an additional block corresponding to the main block; wherein the preset second data amount is smaller than the preset first data amount.

[0057] For each primary block, the starting position information of the primary block and a preset first data volume, the end position information of the primary block is determined. Then, with the end position information as a starting point, the file is divided according to a preset second data volume to obtain additional blocks corresponding to the primary block. The preset second data volume is smaller than the preset first data volume and is greater than the data volume of a complete record.

[0058] In this application, loading the loading data includes:

[0059] The data reading operation, the data parsing operation and the data recording operation in the database are decoupled and asynchronous, and the loading data is loaded according to the decoupled and asynchronous strategy.

[0060] For example, the first part of the data can be read first. Then, while the first part is being parsed, the second part can be read. While the first part is being recorded in the database, the second part can be parsed and the third part can be read. This saves file reading time, as file records read by a single process can be parsed and recorded in parallel by multiple processes, improving file loading performance.

[0061] In this application, the method further includes:

[0062] Dividing the file to be loaded according to a preset first data volume to obtain respective primary blocks of the file, and recording identification information of the respective primary blocks;

[0063] When a data loading exception prompt message is received, the identification information of the abnormal main block carried in the prompt message is obtained; according to the identification information of the abnormal main block, the abnormal main block is reallocated to the corresponding process, and the loading data corresponding to the abnormal main block is reloaded.

[0064] After the main blocks are divided, the present application records the identification information of each main block. For example, the electronic device can number and sort the main blocks, and the sequence number of each main block can be considered as the identification information of the main block.

[0065] During the data loading process, if a data loading exception occurs, including any exception in the data reading operation exception, data parsing operation exception and data recording operation exception, a data loading exception prompt message will be sent to the electronic device, and the data loading exception prompt message will carry the identification information of the abnormal main block. When the electronic device receives the data loading exception prompt message, it determines the abnormal main block based on the identification information of the abnormal main block carried in the prompt message, and then reallocates the abnormal main block to the corresponding process, and reloads the loading data corresponding to the abnormal main block. The present application does not need to reload all the data in the file when the data loading exception occurs, but only needs to reload the data of the main file with the exception, thereby improving the efficiency of emergency handling of exceptions.

[0066] In the present application, before dividing the files to be loaded according to the preset first data volume, the method further includes:

[0067] Randomly number each file that needs to be loaded;

[0068] When a scheduling task is received, the file number interval is determined based on the scheduling value and the total number of processes;

[0069] The files within the file number range are divided into primary blocks in parallel.

[0070] After randomly numbering each file to be loaded, obtain the scheduled value for the scheduled task and determine the file numbering interval based on the scheduled value and the total number of processes. For example, the file numbering interval is ([total number of processes * (current scheduled value - 1) + 1, total number of processes * (scheduled value - 1) + total number of processes]). Files within this file numbering interval are then divided into primary blocks in parallel.

[0071] The method further comprises:

[0072] According to the preset size of each batch of loaded data, and in descending order of the size of the main blocks, the loaded data corresponding to each main block in each file is loaded in batches.

[0073] For example, the order of data size of the Nth batch of main blocks from large to small is: file m block 4: 10M size, file m+1 block 1: 3M size, file m+2 block 3: 3M size, file m+3 block 3: 2M size, and so on.

[0074] The method further comprises:

[0075] For each main block in each file in a single batch, the main blocks are grouped, wherein the difference in the total amount of loaded data corresponding to the main blocks between the groups is less than a preset data amount threshold;

[0076] Assign the primary blocks of each group to the corresponding processes for parallel data loading. The number of groups must be at least the number of processes. Optionally, the number of groups can be the same as the number of processes, or a multiple of the number of processes.

[0077] For example, in batch N, group 01 is file m, block 4: 10M size; group 02 is file m+1, block 1: 3M size, file m+2, block 3: 3M size, file m+3, block 3: 2M size, file m+4, block 3: 2M size; group 03 is file m+5, block 3: 2M size, file m+6, block 3: 2M size, file m+7, block 3: 2M size, file m+8, block 3: 1M size, file m+9, block 3: 0.5M size, file m+10, block 3: 0.5M size.

[0078] The file loading process provided by this application is described in detail below with reference to the accompanying drawings.

[0079] First, the current conventional file loading process is described.

[0080] Figure 2 This is a diagram of the existing technology for loading multiple files concurrently, with a single file being loaded by a single process. Typically, when loading multiple files, application systems use multiple processes concurrently, with each process loading certain files according to specific rules. A single file is not split between multiple processes. When the file transaction volume is low, the loading time is very short. However, when the file transaction volume is high, the loading time can be significantly delayed. Furthermore, if an abnormal interruption occurs during the loading of a large file, the entire file must be reloaded, and the emergency response time cannot meet business requirements.

[0081] Figure 3 This is a schematic diagram of the mode in which multiple processes load a single file concurrently in the prior art. Multiple processes loading the same file alleviates the time drag caused by a single process loading a large file to a certain extent, but it cannot completely avoid the drag phenomenon. In addition, multiple processes read all the data of the same file at the same time, and filter out the rows that do not need to be processed based on the current process ID (customized, not system-assigned), which greatly increases the storage I / O load. The higher the concurrency number, the more likely it is to cause a storage I / O bottleneck. If an abnormal interruption occurs during the loading process, all files loaded by the failed process must be reloaded. The emergency handling process is complicated and prone to multiple loading and missing loading records.

[0082] Existing technologies suffer from significant load delays when a single file's transaction volume is high. Abnormal interruptions during the load process complicate emergency response and increase the time and cost of reloading. Concurrently loading the same file by multiple processes increases the storage I / O load, impacting file loading performance. These issues are particularly pronounced when processing massive file volumes, and higher concurrency increases the likelihood of storage I / O bottlenecks.

[0083] The file loading method provided in this application mainly includes logical block strategy, load balancing strategy, read-write separation strategy, breakpoint resumption strategy and multi-batch processing strategy.

[0084] Logical block strategy: without traversing the file content, the loaded file is marked as blocks, and the size of each block is as similar as possible. Each logical block consists of two parts: main block + additional block, where the main block is the logical block key information calculated according to the pre-set block size: file name, logical block sequence number, starting position, block size. The additional block contains a piece of content in the same file after the end of the main block. The size of this block content is set to be at least the size of a record. The key information of the additional block is: file name, main logical block sequence number, starting position, block size. The main block + additional block method can effectively solve the situation where a record is split into two main logical blocks: the main block starts processing from the first new record, and the additional block only adds the content before the first new record to the main block for processing.

[0085] Load balancing strategy: After logically partitioning the file to be loaded, one or more partitions are combined into a single task to be processed. Taking into account the application system's resource allocation (storage, memory, CPU, etc.), the number of processes for concurrent processing of different tasks can be appropriately allocated. Furthermore, the logical partitioning of the file already records the starting position of each partition within the file and the size of the block to be loaded. During file loading, the offset is used to directly locate the data block to be loaded, avoiding storage I / O bottlenecks. When encountering scenarios with massive amounts of small files, combining partitions effectively improves concurrent loading performance.

[0086] Read-write separation strategy: decouples and asynchronously processes file reading, record parsing, and database recording, saving file reading time. File records read by a single process can be parsed and recorded concurrently by multiple processes, improving file loading task processing performance.

[0087] Breakpoint resume strategy: During the file loading process, the processing and loading of each logical block will be recorded. Once an exception occurs during the processing of a task, no excessive manual intervention is required. Simply resend the file loading task and only the abnormal logical block will be reloaded, thus achieving breakpoint resumption.

[0088] Multi-batch processing strategy: Loading tasks are processed in batches, and subsequent business processing can be carried out without waiting for all files to be loaded, thereby improving the overall business processing efficiency of the application system.

[0089] The entire file loading process is divided into three steps: the first step: file loading preprocessing, that is, numbering the files to be loaded, and then performing logical block division concurrently; the second step: batching and grouping the logical blocks in the previous step; the third step: loading files in multiple batches and with high concurrency.

[0090] Figure 4 The document numbering flow chart provided for this application, such as Figure 4 As shown, the N files to be loaded are randomly numbered from 1 to N. After this process is completed, the next step is to schedule tasks according to values ​​from 1 to N / M+1 (M is the total number of processes), and concurrent file processing is logically divided into blocks. The maximum number of files each process needs to process is N / M. The numbered file information record mainly includes: file number, file name, and file directory.

[0091] Figure 5 The file logic block flow chart provided for this application is as follows: Figure 5 As shown, the file is logically partitioned and processed concurrently. After the partitioning process receives the scheduling task, it calculates the file number interval to be processed based on the scheduling value ([total number of processes * (current scheduling value - 1) + 1, total number of processes * (scheduling value - 1) + total number of processes]). It then retrieves the file information to be processed from the file loading information table, reads the size of the file to be loaded, and divides the file into multiple logical main blocks and additional blocks: Main block: file name, logical block sequence number, starting position, and block size. Additional block: file name, main logical block sequence number, starting position (i.e., the end of the main logical block), and block size. If there are not enough blocks, a separate main block is recorded (no additional blocks). Files that do not meet the block size are also recorded as separate blocks (no additional blocks). The file information record after partitioning mainly includes: file number, file name, file directory, main block sequence number, main block starting position, main block size, additional block starting position, and additional block size.

[0092] Figure 6 The file logic block and batch flow chart provided for this application is as follows: Figure 6 As shown, logical block information is processed in batches. Logical block information for each file is retrieved from the file block information table and sorted in descending order by size. The logical block sizes are accumulated for each item, and multiple logical blocks are labeled as the same batch, keeping them as close to the maximum size of each batch as possible. During batching, the batch information for each logical block is recorded: batch ID, file name, file directory, primary block sequence number, primary block starting position, primary block size, additional block starting position, and additional block size.

[0093] Figure 6 In the file block information table, the file information after segmentation is stored, including file 1 block 1: 10M size; file 1 block 2: 10M size, file 1 block 3: 10M size, file 1 block 4: 10M size, file 1 block 5: 4M size; file 2 block 1: 10M size, file 2 block 2: 10M size, file 2 block 3: 6M size; file 3 block 1: 10M size, file 3 block 2: 10M size, file 3 block 3: 10M size, file 3 block 4: 10M size, file 3 block 5: 5M size; file 4 block 1: 1M size; file 5 block 1: 9M size; file 6 block 1: 3M size.

[0094] Get the file block information table in descending order according to the block size, and determine whether it has been retrieved. If it has been retrieved, the block batch information is recorded in the database. If it has not been retrieved, get the block file information and the accumulated block size to determine whether it exceeds the batch size (for example, 30M). If it does not exceed, get another block file information and the accumulated block size, and continue to determine whether it exceeds the batch size. If it exceeds, put the last accumulated block into the next batch, record the current batch information, batch number, all block information, batch number + 1, and enter the next batch block size accumulation.

[0095] The file block and batch information table stores file information after block and batch division, including batch 01: file 1 block 1: 10M size, file 1 block 2: 10M size, file 1 block 3: 10M size; batch 02: file 1 block 4: 10M size, file 2 block 1: 10M size, file 2 block 2: 10M size; batch 03: file 3 block 1: 10M size, file 3 block 2: 10M size, file 3 block 3: 10M size; batch 04: file 3 block 4: 10M size, file 5 block 1: 9M size, file 2 block 3: 6M size, file 3 block 5: 5M size; batch 05: file 1 block 5: 4M size, file 6 block 1: 3M size, file 4 block 1: 1M size.

[0096] Figure 7 The application provides a flowchart of the logical block grouping process for files. For logical blocks of the same batch, they are grouped according to the number of concurrent processes during file loading. Grouping involves assigning multiple logical blocks to the same group, which is processed by the same file loading process. The total size of the file blocks within the group that each process needs to process is kept as the same as possible, which effectively solves the tailing problem. When grouping batches, the following information is recorded for each group: batch ID, group ID, file name, file directory, primary block sequence number, primary block starting position, primary block size, additional block starting position, and additional block size.

[0097] like Figure 7 As shown, the Nth batch includes file m, block 4:10M size, file m+1, block 1:3M size, file m+2, block 3:3M size, file m+3, block 3:2M size, file m+4, block 3:2M size, file m+5, block 3:1M size, file m+6, block 3:1M size, file m+7, block 3:0.5M size, file m+8, block 3:0.3M size, file m+9, block 3:0.3M size, and file m+10, block 3:0.1M size.

[0098] Get the file block and batch information table in descending order according to the block size, and judge whether it has been retrieved. If it has been retrieved, the block group information is recorded in the database. If it has not been retrieved, get a block file information and the cumulative block size, and judge whether it exceeds the number of group records (for example, 10M). If it does not exceed, get another block file information and the cumulative block size, and continue to judge whether it exceeds the number of group records. If it exceeds, put the last accumulated block into the next group, record the current group information: group id, all block information, group id+1, and enter the next group block size accumulation.

[0099] The file block and batch grouping information table stores the grouped file information after block and batch grouping, including in batch N, group 01: file m block 4: 10M size; group 02: file m+1 block 1: 3M size, file m+2 block 3: 3M size, file m+3 block 3: 2M size, file m+4 block 3: 2M size; group 03: file m+5 block 3: 1M size, file m+6 block 3: 1M size, file m+7 block 3: 0.5M size, file m+8 block 3: 0.3M size, file m+9 block 3: 0.3M size, file m+10 block 3: 0.1M size.

[0100] Figure 8 This is a schematic diagram of the logical block data reading solution provided by this application. The main block starts processing from the first new record, and the additional block only adds the content before the first new record to the main block for processing. Figure 8 As shown, a total of 5 records are processed when reading from main block 1 to additional block 1, and a total of 4 records are processed when reading from main block 2 to additional block 2.

[0101] Figure 9The overall flow chart of concurrent file loading provided in this application performs concurrent file logical block loading, and each process needs to process multiple logical blocks within the group. The file block batch grouping information table includes the first batch 01 group logical blocks, the first batch 02 group logical blocks, the first batch... group logical blocks, and the Nth batch X group logical blocks. The file to be loaded and the file block batch grouping information table input file reading processes P1 to Pm, and the message queue is used for asynchronous reading and writing. Record parsing and library recording processes 1 to N perform database file data storage table (Nth batch table). After all reading is completed, record parsing and library recording processing are completed, the logical block status is set to completion, and the file block batch grouping information table (update status field) is displayed.

[0102] Figure 10 The file loading and reading asynchronous processing flow chart provided by this application shows that the file reading process sends the read file records to the message queue. When the message queue is saturated, it stops writing file records and waits until it can be written. If the wait timeout is reached, it exits abnormally. The record parsing and database recording process is responsible for continuously obtaining file records from the message queue for processing.

[0103] Figure 11 The breakpoint resumption flowchart provided in this application sets the logical block loading status of the file in the file loading status information table to processing when each logical block starts to be read. Thereafter, the logical block record is written to the message queue 2 for record parsing and library recording process (referred to as processing process) to read and process. When all the logical block data are read, the logical block loading status of the file in the file loading status information table is set to read completed. At the same time, the reading process needs to send multiple (the number of messages is consistent with the number of processing processes) read completion messages to the message queue 1. After the processing process obtains its own read completion message from the message queue 1, it first completes the processing of all the logical block data, and then sends the logical block processing completion message to the message queue 1. The logical block reading process obtains the logical block processing completion messages of all processing processes from the message queue 1. The logical block reading process sets the logical block loading status of the file in the file loading status information table to processing completed.

[0104] If a logical block processing exception occurs, the file loading task needs to be re-initiated. For logical block processing status that is not completed, the logical block reading process will first clean up the database file data storage table data according to the logical block (index), and then start the file reading operation, thereby achieving rapid resumption of loading processing after abnormal interruption.

[0105] Figure 12 This is a schematic diagram of the file loading device provided in this application, which includes:

[0106] A partitioning module 111 is configured to partition the file to be loaded according to a preset first data size to obtain primary blocks of the file, and record starting position information of the primary blocks in the file;

[0107] The determination module 112 is used to obtain and determine the number of file loading processes based on the configuration information of the application system resources;

[0108] The loading module 113 is used to allocate the various main blocks to the corresponding processes according to the load balancing strategy. When loading the corresponding main blocks in the process, the loading data corresponding to the main blocks is determined according to the starting position information of the main blocks and the preset first data volume, and the loading data is loaded.

[0109] The partitioning module 111 is further configured to determine, for each primary block, an additional block corresponding to the primary block, wherein the additional block includes data after the end of the primary block in the file;

[0110] The loading module 113 is specifically used to allocate the respective main blocks and their corresponding additional blocks to the corresponding processes according to the load balancing strategy. When loading the corresponding main block in the process, the first loading data corresponding to the main block is determined according to the starting position information of the main block and the preset first data volume; the second loading data before the first complete record in the additional block corresponding to the main block is determined; the second loading data is supplemented to the main block, and the first loading data and the second loading data are loaded.

[0111] The partitioning module 111 is specifically used to determine the end position information of each main block, and to partition the file according to a preset second data amount based on the end position information to obtain additional blocks corresponding to the main blocks; wherein the preset second data amount is smaller than the preset first data amount.

[0112] The loading module 113 is specifically used to decouple and asynchronously perform data reading operations, data parsing operations, and data recording operations into a database, and load the loading data according to the decoupled and asynchronous strategies.

[0113] The loading module 113 is further used to divide the file to be loaded according to a preset first data amount to obtain the main blocks of the file, and record the identification information of the main blocks; when receiving a data loading abnormality prompt message, obtain the identification information of the abnormal main block carried in the prompt message; according to the identification information of the abnormal main block, reallocate the abnormal main block to the corresponding process, and reload the loading data corresponding to the abnormal main block.

[0114] The partitioning module 111 is also used to perform random numbering on each file to be loaded; when a scheduling task is received, the file numbering interval is determined according to the scheduling value and the total number of processes; and the files within the file numbering interval are divided into primary blocks in parallel.

[0115] The loading module 113 is specifically configured to load the loading data corresponding to each main block in each file in batches according to the preset size of each batch of loading data and in the order of the main block data size from large to small.

[0116] The loading module 113 is specifically used to group the main blocks in each file in a single batch, wherein the difference in the total amount of loaded data corresponding to the main blocks between the groups is less than a preset data amount threshold; and allocate the main blocks of each group to the corresponding process for parallel data loading.

[0117] The present application also provides an electronic device, such as Figure 13 As shown, it includes: a processor 121, a communication interface 122, a memory 123 and a communication bus 124, wherein the processor 121, the communication interface 122, and the memory 123 communicate with each other through the communication bus 124;

[0118] The memory 123 stores a computer program, and when the program is executed by the processor 121 , the processor 121 performs any of the above method steps.

[0119] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0120] The communication interface 122 is used for communication between the electronic device and other devices.

[0121] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0122] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0123] The present application also provides a computer storage readable storage medium, which stores a computer program that can be executed by an electronic device. When the program runs on the electronic device, the electronic device implements any of the above method steps when executing.

[0124] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0125] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A file loading method, characterized in that: The method comprises: Dividing the file to be loaded according to a preset first data amount to obtain respective main blocks of the file, and recording starting position information of the respective main blocks in the file; Obtain and determine the number of file loading processes based on the configuration information of application system resources; Allocating each of the master blocks to a corresponding process according to a load balancing strategy; when loading the corresponding master block in the process, determining loading data corresponding to the master block based on the starting position information of the master block and the preset first data volume; and loading the loading data; The method further comprises: For each of the primary blocks, determining an additional block corresponding to the primary block, wherein the additional block includes data after the end of the primary block in the file; Allocating the respective primary blocks and the corresponding additional blocks to corresponding processes according to a load balancing strategy, and when loading the corresponding primary blocks in the processes, determining first loading data corresponding to the primary blocks according to the starting position information of the primary blocks and the preset first data volume; Determine the second loaded data before the first complete record in the additional block corresponding to the main block; Supplement the second loading data to the main block, and load the first loading data and the second loading data; For each of the main blocks, determining the additional block corresponding to the main block includes: For each of the main blocks, determine the end position information of the main block, take the end position information as the starting point, divide the file according to a preset second data amount, and obtain an additional block corresponding to the main block; wherein the preset second data amount is smaller than the preset first data amount.

2. The method according to claim 1, wherein Loading the loading data includes: The data reading operation, the data parsing operation and the data recording operation in the database are decoupled and asynchronous, and the loading data is loaded according to the decoupled and asynchronous strategy.

3. The method according to claim 1, wherein The method further comprises: Dividing the file to be loaded according to a preset first data volume to obtain respective primary blocks of the file, and recording identification information of the respective primary blocks; When a data loading exception prompt message is received, the identification information of the abnormal main block carried in the prompt message is obtained; according to the identification information of the abnormal main block, the abnormal main block is reallocated to the corresponding process, and the loading data corresponding to the abnormal main block is reloaded.

4. The method according to claim 1, wherein Before dividing the files to be loaded according to the preset first data volume, the method further includes: Randomly number each file that needs to be loaded; When a scheduling task is received, the file number interval is determined based on the scheduling value and the total number of processes; The files within the file number range are divided into primary blocks in parallel.

5. The method according to claim 4, wherein The method further comprises: According to the preset size of each batch of loaded data, and in descending order of the size of the main blocks, the loaded data corresponding to each main block in each file is loaded in batches.

6. The method according to claim 5, wherein The method further comprises: For each main block in each file in a single batch, the main blocks are grouped, wherein the difference in the total amount of loaded data corresponding to the main blocks between the groups is less than a preset data amount threshold; Allocate the main blocks of each group to the corresponding process for parallel data loading.

7. A file loading device, characterized in that: The device comprises: a partitioning module, configured to partition the file to be loaded according to a preset first data volume to obtain respective primary blocks of the file, and record starting position information of the respective primary blocks in the file; A determination module is used to obtain and determine the number of file loading processes based on the configuration information of the application system resources; a loading module, configured to allocate the respective master blocks to corresponding processes according to a load balancing strategy, and when loading the corresponding master block in the process, determine the loading data corresponding to the master block based on the starting position information of the master block and the preset first data volume, and load the loading data; The partitioning module is further configured to determine, for each of the primary blocks, an additional block corresponding to the primary block, wherein the additional block includes data after the end of the primary block in the file; The loading module is specifically configured to allocate the respective primary blocks and the corresponding additional blocks to corresponding processes according to a load balancing strategy, and when loading the corresponding primary block in the process, determine first loading data corresponding to the primary block based on the starting position information of the primary block and the preset first data volume; determine second loading data before the first complete record in the additional block corresponding to the primary block; supplement the second loading data to the primary block, and load the first loading data and the second loading data; The partitioning module is specifically used to determine the end position information of each main block, and to partition the file according to a preset second data amount based on the end position information to obtain additional blocks corresponding to the main blocks; wherein the preset second data amount is smaller than the preset first data amount.

8. The device according to claim 7, wherein The loading module is specifically used to decouple and asynchronously perform data reading operations, data parsing operations, and data recording operations into a database, and load the loading data according to the decoupled and asynchronous strategies.

9. The device according to claim 7, wherein The loading module is further configured to divide the file to be loaded according to a preset first data volume to obtain respective primary blocks of the file, and record identification information of the respective primary blocks; and upon receiving a data loading abnormality prompt message, obtain identification information of the abnormal primary block carried in the prompt message; According to the identification information of the abnormal main block, the abnormal main block is reallocated to the corresponding process, and the loading data corresponding to the abnormal main block is reloaded.

10. The device according to claim 7, wherein The partitioning module is also used to perform random numbering on each file that needs to be loaded; when a scheduling task is received, the file numbering interval is determined according to the scheduling value and the total number of processes; and the files within the file numbering interval are divided into primary blocks in parallel.

11. The device according to claim 10, wherein The loading module is specifically used to load the loading data corresponding to each main block in each file in batches according to the preset size of each batch of loading data and in the order of the main block data size from large to small.

12. The device according to claim 11, wherein The loading module is specifically used to group the main blocks in each file in a single batch, wherein the difference in the total amount of loaded data corresponding to the main blocks between the groups is less than a preset data amount threshold; and allocate the main blocks of each group to the corresponding process for parallel data loading.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 6 are implemented.

Images