File loading method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to the technical field of big data, and particularly discloses a file loading method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: splitting a to-be-loaded file to obtain a plurality of sub-files; performing fragmentation processing on each sub-file to obtain file fragmentation information of each sub-file; generating a sub-task corresponding to the file fragment information of each sub-file according to the file fragment information; obtaining a plurality of sub-tasks corresponding to each sub-file based on the sub-tasks corresponding to the file fragment information of each sub-file; and calling concurrent multiple threads, obtaining target data corresponding to each subtask from the subfile corresponding to each subtask based on the initial position information and the end position information included in each subtask, performing corresponding data processing on the target data to obtain processed data, and sending the processed data to the server. And loading the processed data into a target data table corresponding to the to-be-loaded file. By adopting the method, the file loading efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a file loading method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In the banking and other financial industries, as well as various data-intensive business scenarios, there is a need for rapid loading and processing of massive amounts of data files. As business scales up, the size of data files continues to grow. How to efficiently load large files into target data tables has become a crucial issue in enterprise information system construction. Efficient file loading ensures timely data entry, providing strong support for business analysis and decision support.

[0003] Traditional techniques for loading large files often involve first loading them into an intermediate table, then processing each record individually before writing them to the target table. However, this process is lengthy and inefficient. Therefore, a solution to address this inefficiency is urgently needed. Summary of the Invention

[0004] Therefore, it is necessary to provide a file loading method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve file loading efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a file loading method. The method includes:

[0006] Obtain the file to be loaded, and split the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded;

[0007] According to a preset line number threshold, each sub-file is split into segments to obtain file segmentation information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when splitting the sub-file; each file segmentation information includes start position information and end position information, which are used to represent the start splitting position and end splitting position in the corresponding sub-file, respectively;

[0008] Based on the file segmentation information of each sub-file, a sub-task corresponding to the file segmentation information of each sub-file is generated; each sub-task includes the start position information and end position information in the corresponding file segmentation information.

[0009] Based on the sub-tasks corresponding to the file fragment information of each sub-file, multiple sub-tasks corresponding to each sub-file are obtained;

[0010] The system invokes concurrent multithreading to retrieve target data corresponding to each subtask from its corresponding subfile, based on the start and end position information of each subtask. It then performs corresponding data processing on the target data to obtain processed data, which is then loaded into the target data table corresponding to the file to be loaded. The target data corresponding to each subtask represents the data in the corresponding subfile that matches the start and end position information.

[0011] In one embodiment, the step of performing fragmentation processing on each sub-file according to a preset line count threshold to obtain file fragmentation information for each sub-file includes:

[0012] Based on the preset line number threshold, each sub-file is segmented to obtain multiple start position information in each sub-file, and end position information associated with each start position information;

[0013] Based on each starting position information and the ending position information associated with each starting position information, multiple file fragment information for each sub-file is generated.

[0014] In one embodiment, the concurrent multithreading, based on the start and end position information included in each subtask, obtains the target data corresponding to each subtask from the subfile corresponding to each subtask, including:

[0015] Obtain the number of processor cores on each compute node in the server cluster;

[0016] The number of concurrent threads is determined based on the number of processor cores in each computing node of the server cluster.

[0017] The number of concurrent threads is called to obtain the target data corresponding to each subtask from the sub-file corresponding to each subtask, based on the start position information and end position information included in each subtask.

[0018] In one embodiment, determining the number of concurrent threads based on the number of processor cores in each computing node of the server cluster includes:

[0019] Obtain the number of processor cores of a single computing node in the server cluster;

[0020] The number of concurrent threads on a single computing node is determined based on a preset ratio coefficient and the number of processor cores on that single computing node.

[0021] The number of concurrent threads is determined based on the total number of computing nodes in the server cluster and the number of concurrent threads on a single computing node.

[0022] In one embodiment, after loading the processed data into the target data table corresponding to the file to be loaded, the method further includes:

[0023] If the processed data is successfully loaded into the target data table, update the processing status information of the subtask corresponding to the processed data;

[0024] If the processing status information of all subtasks corresponding to the multiple sub-files indicates that the processing is complete, then the multiple sub-files are cleaned up.

[0025] In one embodiment, the step of splitting the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded includes:

[0026] Obtain the file size information of the file to be loaded;

[0027] Based on the file size information and the preset file size threshold, the splitting information of the file to be loaded is determined;

[0028] Based on the splitting information, the file to be loaded is split to obtain the multiple sub-files.

[0029] Secondly, this application also provides a document loading device. The device includes:

[0030] The file acquisition module is used to acquire the file to be loaded, split the file to be loaded, and obtain multiple sub-files corresponding to the file to be loaded.

[0031] The file processing module is used to perform fragment processing on each sub-file according to a preset line number threshold to obtain file fragment information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when performing fragment processing on the sub-file; each file fragment information includes start position information and end position information, the start position information and the end position information are used to represent the start split position and end split position in the corresponding sub-file, respectively;

[0032] The task generation module is used to generate subtasks corresponding to the file segment information of each sub-file based on the file segment information of each sub-file; each subtask includes the start position information and end position information in the corresponding file segment information;

[0033] The task determination module is used to obtain multiple sub-tasks corresponding to each sub-file based on the sub-tasks corresponding to the file fragment information of each sub-file.

[0034] The data loading module is used to invoke concurrent multi-threading to obtain target data corresponding to each sub-task from the sub-file corresponding to each sub-task based on the start position information and end position information included in each sub-task. It then performs corresponding data processing on the target data to obtain processed data and loads the processed data into the target data table corresponding to the file to be loaded. The target data corresponding to each sub-task represents the data in the corresponding sub-file that matches the start position information and the end position information.

[0035] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0036] Obtain the file to be loaded, and split the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded;

[0037] According to a preset line number threshold, each sub-file is split into segments to obtain file segmentation information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when splitting the sub-file; each file segmentation information includes start position information and end position information, which are used to represent the start splitting position and end splitting position in the corresponding sub-file, respectively;

[0038] Based on the file segmentation information of each sub-file, a sub-task corresponding to the file segmentation information of each sub-file is generated; each sub-task includes the start position information and end position information in the corresponding file segmentation information.

[0039] Based on the sub-tasks corresponding to the file fragment information of each sub-file, multiple sub-tasks corresponding to each sub-file are obtained;

[0040] The system invokes concurrent multithreading to retrieve target data corresponding to each subtask from its corresponding subfile, based on the start and end position information of each subtask. It then performs corresponding data processing on the target data to obtain processed data, which is then loaded into the target data table corresponding to the file to be loaded. The target data corresponding to each subtask represents the data in the corresponding subfile that matches the start and end position information.

[0041] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0042] Obtain the file to be loaded, and split the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded;

[0043] According to a preset line number threshold, each sub-file is split into segments to obtain file segmentation information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when splitting the sub-file; each file segmentation information includes start position information and end position information, which are used to represent the start splitting position and end splitting position in the corresponding sub-file, respectively;

[0044] Based on the file segmentation information of each sub-file, a sub-task corresponding to the file segmentation information of each sub-file is generated; each sub-task includes the start position information and end position information in the corresponding file segmentation information.

[0045] Based on the sub-tasks corresponding to the file fragment information of each sub-file, multiple sub-tasks corresponding to each sub-file are obtained;

[0046] The system invokes concurrent multithreading to retrieve target data corresponding to each subtask from its corresponding subfile, based on the start and end position information of each subtask. It then performs corresponding data processing on the target data to obtain processed data, which is then loaded into the target data table corresponding to the file to be loaded. The target data corresponding to each subtask represents the data in the corresponding subfile that matches the start and end position information.

[0047] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0048] Obtain the file to be loaded, and split the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded;

[0049] According to a preset line number threshold, each sub-file is split into segments to obtain file segmentation information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when splitting the sub-file; each file segmentation information includes start position information and end position information, which are used to represent the start splitting position and end splitting position in the corresponding sub-file, respectively;

[0050] Based on the file segmentation information of each sub-file, a sub-task corresponding to the file segmentation information of each sub-file is generated; each sub-task includes the start position information and end position information in the corresponding file segmentation information.

[0051] Based on the sub-tasks corresponding to the file fragment information of each sub-file, multiple sub-tasks corresponding to each sub-file are obtained;

[0052] The system invokes concurrent multithreading to retrieve target data corresponding to each subtask from its corresponding subfile, based on the start and end position information of each subtask. It then performs corresponding data processing on the target data to obtain processed data, which is then loaded into the target data table corresponding to the file to be loaded. The target data corresponding to each subtask represents the data in the corresponding subfile that matches the start and end position information.

[0053] The aforementioned file loading method, apparatus, computer device, computer-readable storage medium, and computer program product acquire a file to be loaded, split the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded; perform fragmentation processing on each sub-file according to a preset line number threshold to obtain file fragmentation information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when performing fragmentation processing on the sub-file; each file fragmentation information includes start position information and end position information, the start position information and end position information being used to represent the start splitting position and end splitting position in the corresponding sub-file, respectively; and generate file fragmentation information for each sub-file based on the file fragmentation information of each sub-file. The corresponding subtasks; each subtask includes start and end position information from the corresponding file segment information; based on the subtasks corresponding to the file segment information of each subfile, multiple subtasks corresponding to each subfile are obtained; concurrent multithreading is invoked, based on the start and end position information included in each subtask, to obtain the target data corresponding to each subtask from the subfile corresponding to each subtask, to perform corresponding data processing on the target data, to obtain processed data, and to load the processed data into the target data table corresponding to the file to be loaded; the target data corresponding to each subtask is used to represent the data that matches the start and end position information in the corresponding subfile.

[0054] The beneficial effects of this application are as follows: Splitting the file to be loaded into multiple sub-files facilitates the decomposition of large files into smaller ones, thereby increasing the concurrent processing capability of file loading; Slicing each sub-file according to a preset line count threshold yields file fragment information for each sub-file, allowing for further refinement of each sub-file into multiple data segments, thus enabling finer-grained task division; Generating corresponding subtasks based on the file fragment information of each sub-file, with each subtask including start and end position information, clarifies the data range to be processed by each subtask, facilitating precise task allocation and independent data processing; Calling concurrent multithreading to retrieve target data from the corresponding sub-files based on the start and end position information of each subtask allows multiple threads to process different subtasks simultaneously, fully utilizing computing resources and achieving efficient concurrent processing; Splitting the file to be loaded into multiple sub-files, then further segmenting each sub-file into multiple subtasks, and using concurrent multithreading to independently process each subtask avoids the efficiency bottleneck of single-threaded processing of large files, thus improving file loading efficiency. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a file loading method in one embodiment;

[0057] Figure 2 This is a flowchart illustrating the steps for generating file fragment information in one embodiment;

[0058] Figure 3 This is a flowchart illustrating the steps for obtaining target data corresponding to each subtask in one embodiment;

[0059] Figure 4 This is a flowchart illustrating the file loading method in another embodiment;

[0060] Figure 5 This is a flowchart illustrating the file loading method in yet another embodiment;

[0061] Figure 6 This is a structural block diagram of a file loading device in one embodiment;

[0062] Figure 7This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that in the embodiments of this application, certain existing solutions in the industry, such as software, components, and models, may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations. The acquisition, transmission, storage, use and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0066] In one exemplary embodiment, such as Figure 1 As shown, a file loading method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.; the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the method includes the following steps:

[0067] Step S101: Obtain the file to be loaded, split the file to be loaded, and obtain multiple sub-files corresponding to the file to be loaded;

[0068] Step S102: According to the preset line number threshold, each sub-file is split into segments to obtain file segmentation information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when splitting the sub-file; each file segmentation information includes start position information and end position information, which are used to represent the start splitting position and end splitting position in the corresponding sub-file, respectively.

[0069] Step S103: Based on the file segmentation information of each sub-file, generate a sub-task corresponding to the file segmentation information of each sub-file; each sub-task includes the start position information and end position information in the corresponding file segmentation information;

[0070] Step S104: Based on the sub-tasks corresponding to the file fragment information of each sub-file, obtain multiple sub-tasks corresponding to each sub-file;

[0071] Step S105: Invoke concurrent multithreading, based on the start position information and end position information included in each subtask, obtain the target data corresponding to each subtask from the sub-file corresponding to each subtask, perform corresponding data processing on the target data to obtain processed data, and load the processed data into the target data table corresponding to the file to be loaded; the target data corresponding to each subtask is used to represent the data that matches the start position information and end position information in the corresponding sub-file.

[0072] The file to be loaded can be a large file that requires data loading and processing.

[0073] Splitting can be a command-line operation to split a large file into multiple smaller files.

[0074] Subfiles can be smaller files obtained by splitting the file to be loaded. Loading a large file becomes loading a smaller file, which increases concurrency.

[0075] The preset line number threshold can be a fixed number of lines referenced when splitting subfiles, and this number can be 500 lines.

[0076] The segmentation process can be an operation that traverses the physical location of sub-files and divides them into segments according to a preset line count threshold.

[0077] The file fragmentation information can be the start and end positions of each fragment obtained after each subfile is fragmented according to a preset line number threshold.

[0078] The byte offset can be a physical location identifier used to determine the fragment location in a subfile.

[0079] The starting position information can be the starting split position of each piece of data in the corresponding sub-file.

[0080] The end position information can be the end split position of each piece of data in the corresponding sub-file.

[0081] Subtasks can be processing tasks generated based on the file segmentation information of each subfile, and each subtask only processes the segment of file corresponding to that task.

[0082] Concurrent multithreading can refer to multiple threads started on each machine in the cluster, and the number of threads can be 75% of the number of processors.

[0083] The target data can be data that matches the start and end position information in the corresponding sub-file.

[0084] Data processing can be the logical processing performed on the target data.

[0085] The processed data can be the data obtained after performing corresponding data processing on the target data.

[0086] The target data table can be the data table corresponding to the file to be loaded, used to store the processed data.

[0087] Optionally, the terminal obtains the file to be loaded, uses a file splitting command to split the file to be loaded into multiple sub-files, and performs fragment processing on each sub-file according to a preset line count threshold. It iterates through the physical location of each sub-file, divides it into segments according to the preset line count threshold, and records the start and end positions of each segment within the corresponding sub-file, thus obtaining the file fragment information for each sub-file. Based on the file fragment information of each sub-file, a subtask corresponding to the file fragment information of each sub-file is generated. Each subtask includes the start and end positions from the corresponding file fragment information. The generated subtasks are then processed. The data is stored in the subtask table. Based on the file segmentation information of each subfile, multiple subtasks are obtained for each subfile. Concurrent multithreading is invoked, with each thread fetching one subtask from the subtask table. Based on the start and end position information included in each subtask, the target data corresponding to each subtask is obtained from the subfile corresponding to each subtask. The target data is processed accordingly to obtain the processed data. The processed data is then loaded into the target data table corresponding to the file to be loaded. The target data corresponding to each subtask is used to represent the data that matches the start and end position information in the corresponding subfile.

[0088] In the above file loading method, the file to be loaded is obtained, and the file to be loaded is split to obtain multiple sub-files corresponding to the file to be loaded; according to a preset line number threshold, each sub-file is fragmented to obtain file fragment information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when fragmenting the sub-file; each file fragment information includes start position information and end position information, which are used to represent the start and end split positions in the corresponding sub-file, respectively; based on the file fragment information of each sub-file, a subtask corresponding to the file fragment information of each sub-file is generated; each subtask... The task includes the start and end position information from the corresponding file segment information; based on the sub-tasks corresponding to the file segment information of each sub-file, multiple sub-tasks are obtained for each sub-file; concurrent multi-threading is invoked to obtain the target data corresponding to each sub-task from the sub-files corresponding to each sub-task based on the start and end position information included in each sub-task, perform corresponding data processing on the target data, obtain processed data, and load the processed data into the target data table corresponding to the file to be loaded; the target data corresponding to each sub-task is used to represent the data that matches the start and end position information in the corresponding sub-file. The above method has the following beneficial effects: Splitting the file to be loaded into multiple sub-files helps to break down large files into smaller ones, thus increasing the concurrent processing capability of file loading; Slicing each sub-file according to a preset line count threshold to obtain file fragment information for each sub-file allows for further refinement of each sub-file into multiple data segments, thus enabling finer-grained task division; Generating corresponding subtasks based on the file fragment information of each sub-file, with each subtask including start and end position information, helps to clarify the data range to be processed by each subtask, thus facilitating precise task allocation and independent data processing; Calling concurrent multithreading to retrieve target data from the corresponding sub-files based on the start and end position information of each subtask allows multiple threads to process different subtasks simultaneously, thus fully utilizing computing resources and achieving efficient concurrent processing; Splitting the file to be loaded into multiple sub-files, then further segmenting each sub-file into multiple subtasks, and using concurrent multithreading to independently process each subtask helps to avoid the efficiency bottleneck of single-threaded processing of large files, thus improving file loading efficiency.

[0089] In one exemplary embodiment, reference is made to Figure 2 Based on a preset line count threshold, each sub-file is split into segments to obtain file segmentation information for each sub-file, specifically including the following:

[0090] Step S201: According to the preset line number threshold, each sub-file is segmented to obtain multiple start position information in each sub-file, and the end position information associated with each start position information;

[0091] Step S202: Based on each start position information and the end position information associated with each start position information, generate multiple file fragment information for each sub-file.

[0092] Among them, multiple starting position information can be the starting split position of each data segment in the corresponding sub-file after each sub-file is processed into segments.

[0093] The end position information associated with each start position information can be the end segmentation position of the data segment corresponding to each start position information in the corresponding sub-file.

[0094] Optionally, the terminal performs fragmentation processing on each sub-file according to a preset line count threshold, traverses the physical location of each sub-file, determines the fragmentation boundary of each sub-file according to the preset line count threshold, and obtains multiple start position information in each sub-file, as well as end position information associated with each start position information. Each start position information represents the start splitting position of a data segment, and the end position information associated with each start position information represents the end splitting position of the data segment. Based on each start position information and the end position information associated with each start position information, multiple file fragment information for each sub-file is generated. Each file fragment information contains corresponding start position information and end position information, which are used to describe the position range of a data segment in the corresponding sub-file.

[0095] The technical solution provided in this embodiment segments each sub-file according to a preset line count threshold, obtaining multiple start position information and associated end position information for each sub-file. This facilitates further refining each sub-file into multiple data segments according to a fixed line count, thereby enabling more precise task division. By generating multiple file segment information for each sub-file based on each start position information and associated end position information, the precise location range of each data segment within the corresponding sub-file is clarified. This allows subsequent threads to accurately locate and independently process each data segment, improving the concurrent processing capability and overall efficiency of file loading.

[0096] In one exemplary embodiment, reference is made to Figure 3 This involves invoking concurrent multithreading to retrieve the target data for each subtask from its corresponding subfile, based on the start and end position information of each subtask. Specifically, this includes the following:

[0097] Step S301: Obtain the number of processor cores of each computing node in the server cluster;

[0098] Step S302: Determine the number of concurrent threads based on the number of processor cores in each computing node of the server cluster;

[0099] Step S303: Call the thread of the number of concurrent threads, and based on the start position information and end position information included in each subtask, obtain the target data corresponding to each subtask from the sub-file corresponding to each subtask.

[0100] A server cluster can be a cluster system composed of multiple computing nodes, used to perform file loading tasks.

[0101] In this context, a computing node can be any machine in a server cluster (such as a server) used to perform data processing tasks.

[0102] The number of processor cores can be the number of processor cores in each computing node.

[0103] The number of concurrent threads can be the total number of threads used for concurrent processing, determined based on the number of processor cores in each computing node of the server cluster.

[0104] Optionally, the terminal obtains the number of processor cores of each computing node in the server cluster, counts the processor configuration information of each computing node in the server cluster, determines the number of concurrent threads based on the number of processor cores of each computing node in the server cluster, and the number of concurrent threads is the sum of the number of threads determined by each computing node in the server cluster based on its own number of processor cores. The terminal calls the number of concurrent threads, and each thread grabs a subtask from the subtask table. Based on the start position information and end position information included in each subtask, the terminal obtains the target data corresponding to each subtask from the subfile corresponding to each subtask. Each subtask only processes a segment of data in the corresponding subfile.

[0105] The technical solution provided in this embodiment, by obtaining the number of processor cores of each computing node in the server cluster, is beneficial for understanding the computing resource configuration of the server cluster, thereby facilitating the rational planning of the number of concurrent threads; by determining the number of concurrent threads based on the number of processor cores of each computing node in the server cluster, it is beneficial for dynamically adjusting the concurrent processing capability according to the actual hardware resource configuration, thereby facilitating the full utilization of the computing resources of the server cluster; by calling the threads of the number of concurrent threads, based on the start and end position information included in each subtask, to obtain the target data corresponding to each subtask from the subfile corresponding to each subtask, it is beneficial for multiple threads to concurrently process different subtasks in the server cluster, thereby improving the concurrency and processing efficiency of file loading.

[0106] In an exemplary embodiment, the number of concurrent threads is determined based on the number of processor cores of each computing node in the server cluster. Specifically, this includes: obtaining the number of processor cores of a single computing node in the server cluster; determining the number of concurrent threads for a single computing node based on a preset ratio coefficient and the number of processor cores of a single computing node; and determining the total number of concurrent threads based on the total number of computing nodes in the server cluster and the number of concurrent threads for a single computing node.

[0107] The number of processor cores in a single computing node can be the number of processor cores in a single computing node.

[0108] The preset ratio coefficient can be a proportional parameter used to calculate the number of concurrent threads based on the number of processor cores, and this ratio coefficient can be 75%.

[0109] The number of concurrent threads per computing node can be calculated based on a preset ratio coefficient and the number of processor cores per computing node, which is the number of threads that a single computing node should start.

[0110] The total number of compute nodes can be the total number of compute nodes in the server cluster.

[0111] Optionally, the terminal obtains the number of processor cores of a single computing node in the server cluster, queries the processor configuration information of a single computing node, and determines the number of concurrent threads of a single computing node based on a preset ratio coefficient and the number of processor cores of a single computing node. The number of concurrent threads of a single computing node is the product of the number of processor cores of a single computing node and the preset ratio coefficient. Based on the total number of computing nodes in the server cluster and the number of concurrent threads of a single computing node, the number of concurrent threads is determined, which is the product of the total number of computing nodes and the number of concurrent threads of a single computing node.

[0112] The technical solution provided in this embodiment, by obtaining the number of processor cores of a single computing node in a server cluster, helps to understand the hardware configuration of a single computing node, thereby facilitating the rational planning of the number of threads on a single computing node; by determining the number of concurrent threads on a single computing node based on a preset ratio coefficient and the number of processor cores on a single computing node, it helps to avoid resource contention caused by an excessive number of threads, thereby optimizing the concurrent processing performance of a single computing node; by determining the number of concurrent threads based on the total number of computing nodes in the server cluster and the number of concurrent threads on a single computing node, it helps to integrate the concurrency capabilities of all computing nodes in the server cluster, thereby facilitating the full utilization of the overall computing resources of the server cluster and improving the concurrency and overall efficiency of file loading.

[0113] In an exemplary embodiment, after loading the processed data into the target data table corresponding to the file to be loaded, the following steps are also included: if the processed data is successfully loaded into the target data table, update the processing status information of the subtask corresponding to the processed data; if the processing status information of all subtasks corresponding to multiple subfiles indicates that the processing is complete, perform cleanup processing on the multiple subfiles.

[0114] The processing status information can be information used to indicate the processing status of a subtask, which can indicate whether the subtask has been completed.

[0115] The cleanup process can involve deleting multiple sub-files to free up storage space.

[0116] Optionally, after successfully loading the processed data into the target data table, the terminal updates the processing status information of the subtask corresponding to the processed data, updates the processing status information of the subtask corresponding to the processed data to the processing completed status, traverses the processing status information of all subtasks, determines whether all subtasks have been completed, and if the processing status information of all subtasks corresponding to multiple subfiles indicates that the processing is completed, cleans up the multiple subfiles, deletes the multiple subfiles and releases the storage space.

[0117] The technical solution provided in this embodiment, by updating the processing status information of the subtasks corresponding to the processed data after the processed data is successfully loaded into the target data table, is conducive to recording the processing progress of each subtask in real time, thereby facilitating the tracking of the overall completion of file loading; by cleaning up multiple subfiles when the processing status information of all subtasks corresponding to multiple subfiles indicates that the processing is completed, it is conducive to timely releasing storage space and improving the utilization efficiency of storage resources.

[0118] In an exemplary embodiment, the file to be loaded is split to obtain multiple sub-files corresponding to the file to be loaded. Specifically, this includes: obtaining the file size information of the file to be loaded; determining the splitting information of the file to be loaded based on the file size information and a preset file size threshold; and splitting the file to be loaded based on the splitting information to obtain multiple sub-files.

[0119] The file size information can be either the size of the file to be loaded in bytes or the storage capacity.

[0120] The preset file size threshold can be a baseline value for determining whether the file to be loaded needs to be split and the number of splits.

[0121] The splitting information can be related to splitting the file to be loaded, which is determined based on the file size information and a preset file size threshold. This information includes the number of splits and the splitting location.

[0122] Optionally, the terminal obtains the file size information of the file to be loaded, determines the splitting information of the file to be loaded based on the file size information and a preset file size threshold, determines the number of files to be split based on the ratio of the file size information to the preset file size threshold, and determines the splitting position of the file to be loaded, and splits the file to be loaded into multiple sub-files according to the number of splits and the splitting position in the splitting information.

[0123] The technical solution provided in this embodiment, by obtaining the file size information of the file to be loaded, helps to determine whether splitting is necessary; by determining the splitting information of the file to be loaded based on the file size information and a preset file size threshold, it helps to dynamically determine the splitting strategy based on the actual size of the file to be loaded; by splitting the file to be loaded based on the splitting information to obtain multiple sub-files, it helps to split large files into multiple easily processed sub-files, thereby reducing the amount of data processed in a single operation, avoiding memory overflow problems, and thus improving the efficiency of file loading.

[0124] In one embodiment, such as Figure 4 As shown, another file loading method is provided. Taking the application of this method to a terminal as an example, the steps include:

[0125] Step S401: Obtain the file size information of the file to be loaded; determine the splitting information of the file to be loaded based on the file size information and the preset file size threshold; split the file to be loaded based on the splitting information to obtain multiple sub-files.

[0126] Step S402: According to the preset line number threshold, each sub-file is segmented to obtain multiple start position information in each sub-file, as well as the end position information associated with each start position information; based on each start position information and its associated end position information, multiple file segment information for each sub-file is generated.

[0127] The preset line number threshold is used to characterize the byte offset referenced when splitting subfiles; each file split information includes start position information and end position information, which are used to indicate the start split position and end split position in the corresponding subfile, respectively.

[0128] Step S403: Generate subtasks corresponding to the file fragment information of each sub-file based on the file fragment information of each sub-file; obtain multiple subtasks corresponding to each sub-file based on the subtasks corresponding to the file fragment information of each sub-file.

[0129] Each subtask includes the start and end position information from the corresponding file segment information.

[0130] Step S404: Obtain the number of processor cores of a single computing node in the server cluster; determine the number of concurrent threads of a single computing node based on a preset ratio coefficient and the number of processor cores of a single computing node; determine the number of concurrent threads based on the total number of computing nodes in the server cluster and the number of concurrent threads of a single computing node.

[0131] Step S405: Call the thread of the concurrent thread count, and based on the start position information and end position information included in each subtask, obtain the target data corresponding to each subtask from the sub-file corresponding to each subtask.

[0132] The target data for each subtask is used to represent the data that matches the start and end position information in the corresponding sub-file.

[0133] Step S406: Perform corresponding data processing on the target data to obtain processed data, and load the processed data into the target data table corresponding to the file to be loaded.

[0134] Step S407: If the processed data is successfully loaded into the target data table, update the processing status information of the subtask corresponding to the processed data.

[0135] Step S408: If the processing status information of all subtasks corresponding to multiple sub-files indicates that the processing is complete, clean up the multiple sub-files.

[0136] The above file loading method achieves the following beneficial effects: Firstly, splitting the file to be loaded into multiple sub-files facilitates the decomposition of large files into smaller ones, thereby increasing the concurrent processing capability of file loading. Secondly, segmenting each sub-file according to a preset line count threshold to obtain file segmentation information for each sub-file allows for further refinement of each sub-file into multiple data fragments, thus enabling finer-grained task division. Thirdly, generating corresponding subtasks based on the file segmentation information of each sub-file, with each subtask including start and end position information, clarifies the data range to be processed by each subtask, facilitating precise task allocation and independent data processing. Fourthly, calling concurrent multithreading to retrieve target data from the corresponding sub-files based on the start and end position information of each subtask allows multiple threads to process different subtasks simultaneously, fully utilizing computing resources and achieving efficient concurrent processing. Fifthly, splitting the file to be loaded into multiple sub-files, then segmenting each sub-file into multiple subtasks, and using concurrent multithreading to independently process each subtask avoids the efficiency bottleneck of single-threaded processing of large files, thus improving file loading efficiency.

[0137] The following is an example illustrating the file loading method provided in this application. This example demonstrates the application of this method to a terminal.

[0138] Loading large files is a challenging task, often leading to efficiency issues and a high risk of Out of Memory (OOM) errors. The difficulty increases further if complex processing is required during loading. Generally, loading large files involves some degree of processing logic. While data synchronization tools can mitigate OOM errors to some extent, they cannot handle complex processing during loading. They must first load the data into an intermediate table, then process each record in that table before finally loading it into the target table, which doesn't demonstrate high efficiency. Database-provided data loading tools follow a similar approach; although simple, they remain inefficient and prone to high I / O overhead.

[0139] Option 1: A data synchronization tool loads large files into an intermediate table, then processes the intermediate table row by row using a programming language before writing them into the target table. This avoids OutOfMemoryError (OOM) issues during loading, but the process from the intermediate table to the target table is inefficient.

[0140] Option 2: Load the database data into an intermediate table using a database loading tool, and then process it using a stored procedure cursor before writing it to the target table. While this method is simple to develop and avoids OutOfMemoryError (OOM) issues, it is extremely inefficient and time-consuming.

[0141] This embodiment adopts a method of splitting into sub-files and then further splitting into sub-tasks for concurrent loading, which can avoid the OOM problem. At the same time, it can perform complex file processing logic, and can directly load files into the target table. It is simple, efficient, and flexibly configurable.

[0142] In this embodiment, large files are split into smaller files, transforming the loading of large files into loading of smaller files, thus increasing concurrency. The idea of ​​splitting small files into chunks is used, where each chunk records the start and end positions of the file to be processed, generating subtasks. Multi-threaded fetching of subtasks, with each subtask handling only a minimal amount of data, further improves concurrency.

[0143] This embodiment employs a method of splitting the file and simultaneously fragmenting it into subtasks, effectively avoiding the OutOfMemoryError (OOM) problem. It allows for high concurrency processing on the computing platform, resulting in high efficiency. The specific implementation steps are as follows:

[0144] 1. Large file splitting: Use system commands to split files.

[0145] 2. Small file splitting: Use a programming language to traverse the physical location of the file and split it into pieces according to a fixed number of lines (the fixed number of lines can be configured and initialized to 500), and write the start and end positions of each piece into the subtask table.

[0146] 3. Multi-threaded processing: Each machine in the cluster starts multiple threads, with the number of threads being 75% of the number of processors. For example, if there are 16 cores, then 12 concurrent threads are configured. If there are 6 machines in the cluster, it is equivalent to 72 concurrent threads. Each subtask processes only a segment of the file. Programming languages ​​can be used to read the file, perform complex processing, and write it to the target table.

[0147] refer to Figure 5 The large file is split into small files 1, 2 to N. Small file 1 is further split into subtasks 1-1, 1-2 to N-m. Small file 2 is split into subtasks 2-1, 2-2 to N-m. Small file N is split into subtasks N-1, N-2 to Nm. All subtasks form a subtask set, and all subtasks are processed by multithreading.

[0148] The technical solution provided in this embodiment can easily load large files with complex processing, effectively avoid OOM problems, make full use of multiple machines in the cluster, and achieve high concurrency and efficiency. Due to the high concurrency, subtask processing is relatively fast, and the files can be cleaned up after loading, resulting in a very short actual storage time.

[0149] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a file loading apparatus for implementing the file loading method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more file loading apparatus embodiments provided below can be found in the limitations of the file loading method described above, and will not be repeated here.

[0151] In one exemplary embodiment, such as Figure 6 As shown, a file loading device 600 is provided, which may include:

[0152] The file acquisition module 601 is used to acquire the file to be loaded, split the file to be loaded, and obtain multiple sub-files corresponding to the file to be loaded.

[0153] The file processing module 602 is used to perform fragment processing on each sub-file according to a preset line number threshold to obtain file fragment information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when performing fragment processing on the sub-file; each file fragment information includes start position information and end position information, which are used to indicate the start split position and end split position in the corresponding sub-file, respectively;

[0154] The task generation module 603 is used to generate subtasks corresponding to the file segment information of each sub-file based on the file segment information of each sub-file; each subtask includes the start position information and end position information in the corresponding file segment information;

[0155] The task determination module 604 is used to obtain multiple sub-tasks corresponding to each sub-file based on the sub-tasks corresponding to the file fragment information of each sub-file.

[0156] The data loading module 605 is used to invoke concurrent multi-threading. Based on the start and end position information included in each subtask, it obtains the target data corresponding to each subtask from the sub-file corresponding to each subtask, performs corresponding data processing on the target data, obtains the processed data, and loads the processed data into the target data table corresponding to the file to be loaded. The target data corresponding to each subtask is used to represent the data that matches the start and end position information in the corresponding sub-file.

[0157] In an exemplary embodiment, the file processing module 602 is further configured to perform segmentation processing on each sub-file according to a preset line number threshold, to obtain multiple start position information in each sub-file and end position information associated with each start position information; and to generate multiple file segmentation information for each sub-file based on each start position information and the end position information associated with each start position information.

[0158] In an exemplary embodiment, the data loading module 605 is further configured to obtain the number of processor cores of each computing node in the server cluster; determine the number of concurrent threads based on the number of processor cores of each computing node in the server cluster; and call the thread of the number of concurrent threads to obtain the target data corresponding to each subtask from the subfile corresponding to each subtask based on the start position information and end position information included in each subtask.

[0159] In an exemplary embodiment, the data loading module 605 is further configured to obtain the number of processor cores of a single computing node in the server cluster; determine the number of concurrent threads of a single computing node based on a preset ratio coefficient and the number of processor cores of a single computing node; and determine the number of concurrent threads based on the total number of computing nodes in the server cluster and the number of concurrent threads of a single computing node.

[0160] In an exemplary embodiment, the apparatus further includes a file cleaning module, which is used to update the processing status information of the subtask corresponding to the processed data when the processed data is successfully loaded into the target data table; and to clean up the multiple sub-files when the processing status information of all sub-files corresponding to the multiple sub-files indicates that the processing is completed.

[0161] In an exemplary embodiment, the file acquisition module 601 is further configured to acquire file size information of the file to be loaded; determine the splitting information of the file to be loaded based on the file size information and a preset file size threshold; and split the file to be loaded based on the splitting information to obtain multiple sub-files.

[0162] Each module in the aforementioned file loading device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0163] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a file loading method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0164] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0166] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0167] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0168] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A file loading method, characterized in that, The method includes: Obtain the file to be loaded, and split the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded; According to a preset line number threshold, each sub-file is split into segments to obtain file segmentation information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when splitting the sub-file; each file segmentation information includes start position information and end position information, which are used to represent the start splitting position and end splitting position in the corresponding sub-file, respectively; Based on the file segmentation information of each sub-file, a sub-task corresponding to the file segmentation information of each sub-file is generated; each sub-task includes the start position information and end position information in the corresponding file segmentation information. Based on the sub-tasks corresponding to the file fragment information of each sub-file, multiple sub-tasks corresponding to each sub-file are obtained; The system invokes concurrent multithreading to retrieve target data corresponding to each subtask from its corresponding subfile, based on the start and end position information of each subtask. It then performs corresponding data processing on the target data to obtain processed data, which is then loaded into the target data table corresponding to the file to be loaded. The target data corresponding to each subtask represents the data in the corresponding subfile that matches the start and end position information.

2. The method according to claim 1, characterized in that, The step of segmenting each sub-file according to a preset line count threshold to obtain file segmentation information for each sub-file includes: Based on the preset line number threshold, each sub-file is segmented to obtain multiple start position information in each sub-file, and end position information associated with each start position information; Based on each starting position information and the ending position information associated with each starting position information, multiple file fragment information for each sub-file is generated.

3. The method according to claim 1, characterized in that, The concurrent multithreaded invocation, based on the start and end position information included in each subtask, retrieves the target data corresponding to each subtask from the sub-file corresponding to each subtask, including: Obtain the number of processor cores on each compute node in the server cluster; The number of concurrent threads is determined based on the number of processor cores in each computing node of the server cluster. The number of concurrent threads is called to obtain the target data corresponding to each subtask from the sub-file corresponding to each subtask, based on the start position information and end position information included in each subtask.

4. The method according to claim 3, characterized in that, Determining the number of concurrent threads based on the number of processor cores in each computing node of the server cluster includes: Obtain the number of processor cores of a single computing node in the server cluster; The number of concurrent threads on a single computing node is determined based on a preset ratio coefficient and the number of processor cores on that single computing node. The number of concurrent threads is determined based on the total number of computing nodes in the server cluster and the number of concurrent threads on a single computing node.

5. The method according to claim 1, characterized in that, After loading the processed data into the target data table corresponding to the file to be loaded, the process further includes: If the processed data is successfully loaded into the target data table, update the processing status information of the subtask corresponding to the processed data; If the processing status information of all subtasks corresponding to the multiple sub-files indicates that the processing is complete, then the multiple sub-files are cleaned up.

6. The method according to any one of claims 1 to 5, characterized in that, The process of splitting the file to be loaded to obtain multiple sub-files corresponding to the file to be loaded includes: Obtain the file size information of the file to be loaded; Based on the file size information and the preset file size threshold, the splitting information of the file to be loaded is determined; Based on the splitting information, the file to be loaded is split to obtain the multiple sub-files.

7. A document loading device, characterized in that, The device includes: The file acquisition module is used to acquire the file to be loaded, split the file to be loaded, and obtain multiple sub-files corresponding to the file to be loaded. The file processing module is used to perform fragment processing on each sub-file according to a preset line number threshold to obtain file fragment information for each sub-file; the preset line number threshold is used to characterize the byte offset referenced when performing fragment processing on the sub-file; each file fragment information includes start position information and end position information, the start position information and the end position information are used to represent the start split position and end split position in the corresponding sub-file, respectively; The task generation module is used to generate subtasks corresponding to the file segment information of each sub-file based on the file segment information of each sub-file; each subtask includes the start position information and end position information in the corresponding file segment information; The task determination module is used to obtain multiple sub-tasks corresponding to each sub-file based on the sub-tasks corresponding to the file fragment information of each sub-file. The data loading module is used to invoke concurrent multi-threading to obtain target data corresponding to each sub-task from the sub-file corresponding to each sub-task based on the start position information and end position information included in each sub-task. It then performs corresponding data processing on the target data to obtain processed data and loads the processed data into the target data table corresponding to the file to be loaded. The target data corresponding to each sub-task represents the data in the corresponding sub-file that matches the start position information and the end position information.

8. The apparatus according to claim 7, characterized in that, The file processing module is further configured to perform segmentation processing on each sub-file according to the preset line number threshold, to obtain multiple start position information in each sub-file and end position information associated with each start position information; and to generate multiple file segmentation information for each sub-file according to each start position information and the end position information associated with each start position information.

9. The apparatus according to claim 7, characterized in that, The data loading module is also used to obtain the number of processor cores of each computing node in the server cluster; and to determine the number of concurrent threads based on the number of processor cores of each computing node in the server cluster. The number of concurrent threads is called to obtain the target data corresponding to each subtask from the sub-file corresponding to each subtask, based on the start position information and end position information included in each subtask.

10. The apparatus according to claim 9, characterized in that, The data loading module is also used to obtain the number of processor cores of a single computing node in the server cluster; and to determine the number of concurrent threads of a single computing node based on a preset ratio coefficient and the number of processor cores of the single computing node. The number of concurrent threads is determined based on the total number of computing nodes in the server cluster and the number of concurrent threads on a single computing node.

11. The apparatus according to claim 7, characterized in that, The device further includes a file cleaning module, used to update the processing status information of the subtask corresponding to the processed data when the processed data is successfully loaded into the target data table; and to clean up the multiple subfiles when the processing status information of all subtasks corresponding to the multiple subfiles indicates that the processing is completed.

12. The apparatus according to any one of claims 7 to 11, characterized in that, The file acquisition module is further configured to acquire the file size information of the file to be loaded; determine the splitting information of the file to be loaded based on the file size information and a preset file size threshold; and split the file to be loaded based on the splitting information to obtain the multiple sub-files.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images