File merging method and device, electronic equipment and storage medium
Through dynamically determining the buffer size and multi-threading technology, the problem of slow file merging when banks interact with multiple fund companies is solved, rapid merger and stability improvement is achieved, and documents are delivered on time at the regulatory authorities.
Patent Information
- Application Number
- CN202411190037.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
AI Technical Summary
When banks interact with business data with multiple fund companies, file merging is slow, and the waste or overload of system resources caused by unreasonable buffer settings is difficult to meet the regulatory authorities' time requirements.
By combining the current hardware resource information and the information of the files to be merged, the available buffer size is dynamically determined, and multi-threading technology and memory mapping technology are used to merge files to ensure that the buffer size is adapted to changes in the merge task, maximize the utilization of system resources, and avoid Resources are wasted or overloaded.
Improves file merge speed and stability, reduces overall costs, and ensures file mergers are delivered on time before the time specified by the regulator.
Smart Images

Figure CN119088768A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of financial technology, and in particular to a file merging method, device, electronic device and storage medium. Background Art
[0002] Generally, banks and external systems such as fund companies exchange business data in the form of files. If a bank connects with multiple fund institutions, it is necessary to merge the files of different fund companies before sending them to the regulatory authorities.
[0003] As a bank connects with more and more fund companies, the number of files that need to be merged is also increasing, and the size of a single file is also getting larger and larger, up to GB level. Since the merged files need to be delivered before the time specified by the regulatory agency, how to quickly merge these files has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The embodiments of the present application provide a file merging method, device, electronic device and storage medium, which are used to provide a solution for quickly merging files.
[0005] In a first aspect, an embodiment of the present application provides a file merging method, comprising:
[0006] Merging and sorting the acquired multiple files to obtain a merged order of the multiple files;
[0007] Based on the merging order, according to a rule of merging two files at a time, merging the multiple files into one file through multiple threads;
[0008] The two files to be merged are merged each time according to the following steps:
[0009] The available buffer size is determined according to the current hardware resource information and the file information of the two files, and the file contents of the two files are mapped to the memory according to the available buffer size, and the file contents in the memory are merged.
[0010] In some embodiments, the hardware resource information includes a parameter value of at least one hardware parameter, the file information includes a parameter value of at least one file parameter, and determining the available buffer size according to the current hardware resource information and the file information of the two files includes:
[0011] Determining a first weight according to a parameter value of the at least one hardware parameter;
[0012] Determining a second weight according to a parameter value of the at least one file parameter;
[0013] Determine the difference between the preset value and the weight sum as the buffer weight, where the weight sum refers to the sum of the first weight and the second weight;
[0014] The product of the preset cache size and the buffer weight is determined as the available cache size.
[0015] In some embodiments, the parameter value of the at least one hardware parameter includes a memory usage rate and a CPU load rate, and determining the first weight according to the parameter value of the at least one hardware parameter includes:
[0016] The memory usage rate and the CPU load rate are weighted and summed to obtain the first weight.
[0017] In some embodiments, the parameter value of the at least one file parameter includes a file size and a merge order, and determining the second weight according to the parameter value of the at least one file parameter includes:
[0018] Determine at least two initial files corresponding to the two files according to the merge information of the two files;
[0019] Determine a target file size according to the file sizes of the at least two initial files, and determine a target order according to the merging order of the at least two initial files;
[0020] The target file size and the target order are weighted and summed to obtain the second weight.
[0021] In some embodiments, merging and sorting the acquired multiple files to obtain a merged order of the multiple files includes:
[0022] Performing a weighted summation on the file size and the degree of fragmentation of each file to obtain a score for the file;
[0023] Based on the scores of the multiple files, the merging order of the multiple files is determined according to a rule that the scores are positively correlated with the merging order.
[0024] In some embodiments, after merging the file contents in the memory, the method further includes:
[0025] Calculate the MD5 value of the file content of each file in the merged file;
[0026] If the MD5 value corresponding to any file is different from the MD5 value corresponding to the file before the merge, it is determined that the merge is wrong, and the two files are merged again.
[0027] In some embodiments, it also includes:
[0028] After each successful merge, record index information, the index information including the correspondence between each merged file and the merged file, and the line number range of each merged file in the corresponding merged file;
[0029] If it is determined that any merged file is updated, determining a target merged file containing the file and a row number range corresponding to the file in the target merged file according to the index information;
[0030] The file content corresponding to the line number range is deleted from the target merged file, and the updated file is merged into the target merged file.
[0031] In a second aspect, an embodiment of the present application provides a file merging device, including:
[0032] A sorting module is used to merge and sort the acquired multiple files to obtain a merged order of the multiple files;
[0033] A merging module, configured to merge the multiple files into one file through multiple threads based on the merging order and according to a rule of merging two files each time;
[0034] The two files to be merged are merged each time according to the following steps:
[0035] The available buffer size is determined according to the current hardware resource information and the file information of the two files, and the file contents of the two files are mapped to the memory according to the available buffer size, and the file contents in the memory are merged.
[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:
[0037] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned file merging method.
[0038] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the above-mentioned file merging method.
[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned file merging method when executed by a processor.
[0040] In an embodiment of the present application, the acquired multiple files are merged and sorted to obtain the merge order of the multiple files, and then, based on the merge order, the multiple files are merged into one file through multiple threads according to the rule of merging two files each time, wherein the two files to be merged each time are merged according to the following steps: the available buffer size is determined according to the current hardware resource information and the file information of the two files, the file contents of the two files are mapped to the memory according to the available buffer size, and then the file contents in the memory are merged. In this way, each time a merge is performed, the available buffer size is determined in combination with the current hardware resource information and the file information of the two files to be merged, so that the available buffer size can better adapt to the current merge task and change with the change of the merge task, thereby maximizing the use of system resources, avoiding the waste or overload of system resources caused by unreasonable buffer settings, reducing overall costs, and improving the file merging speed and stability, and the file merging speed can be further improved with the help of multi-threading technology and memory mapping technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0042] Figure 1 A flowchart of a file merging method provided in an embodiment of the present application;
[0043] Figure 2 A schematic diagram of a file sorting process provided in an embodiment of the present application;
[0044] Figure 3 A flow chart of a method for determining the available buffer size provided in an embodiment of the present application;
[0045] Figure 4 A schematic diagram of a memory mapping process provided by an embodiment of the present application;
[0046] Figure 5 A schematic diagram of a process for comparing the MD5 values of files before and after merging provided in an embodiment of the present application;
[0047] Figure 6 A schematic diagram of a process for updating a merged file provided in an embodiment of the present application;
[0048] Figure 7 A schematic diagram of a multi-threaded file merging process provided in an embodiment of the present application;
[0049] Figure 8 A schematic diagram of the structure of a file merging device provided in an embodiment of the present application;
[0050] Fig. 9 A schematic diagram of the hardware structure of an electronic device for implementing a file merging method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to provide a solution for quickly merging files, embodiments of the present application provide a file merging method, device, electronic device, and storage medium.
[0052] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.
[0053] In order to facilitate the understanding of this application, the technical terms involved in this application are:
[0054] Memory: It is a component in a computer used to store and access data, including random access memory (RAM) and read-only memory (ROM). It is a temporary storage area for computers to execute programs and process data.
[0055] Memory-Mapped I / O: Map large files into memory to reduce memory usage and improve input / output (I / O) efficiency.
[0056] Sliding Window Algorithm: It is used to adjust the sliding window size according to the available buffer size. During the file merging process, the file size of the read and write buffers is dynamically adjusted with the help of the sliding window.
[0057] Work Stealing Algorithm: A load balancing technique in parallel computing that is used to dynamically distribute workloads among multiple threads or processors. The core idea is that "lazy" threads will steal the work of "busy" threads.
[0058] Generally, banks and external systems such as fund companies exchange business data in the form of files. If a bank connects with multiple fund institutions, it is necessary to merge the files of different fund companies before sending them to the regulatory authorities.
[0059] As a bank connects with more and more fund companies, the number of files that need to be merged is also increasing, and the size of a single file is also getting larger and larger, up to GB level. Since the merged files need to be delivered before the time specified by the regulatory agency, how to quickly merge these files has become a technical problem that needs to be solved urgently.
[0060] In the related art, after waiting for all the files of the fund companies to arrive, the files are merged in the order in which they arrive. The available cache size is fixed during merging, so it is easy to cause buffer overflow under high load conditions and waste resources under low load conditions, resulting in a relatively slow file merging speed. In addition, the related art uses high-performance hardware such as high memory and high CPU to increase the merging speed, and the hardware cost is also relatively high.
[0061] In order to improve the file merging speed and control hardware costs, the embodiment of the present application provides a file merging method, in which, each time a merge is performed, the available buffer size is determined in combination with the current hardware resource information and the file information of the two files to be merged, so that the available buffer size can better adapt to the current merging task and change with the change of the merging task, thereby maximizing the use of system resources, avoiding system resource waste or overload problems caused by unreasonable buffer settings, reducing overall costs, and improving file merging speed and stability. In addition, multi-threading technology and memory mapping technology are used to further improve the file merging speed.
[0062] See also Figure 1 , Figure 1 A flowchart of a file merging method provided in an embodiment of the present application includes the following steps.
[0063] In step 101, the acquired multiple files are merged and sorted to obtain a merge order of the multiple files.
[0064] The multiple files obtained may be files of multiple fund companies, and each file may include multiple records of the corresponding fund companies. Furthermore, the multiple files may arrive at the same time or at different times.
[0065] In specific implementation, after obtaining a file, the file size can be calculated, and the file fragmentation degree can be calculated with the help of a function for calculating the fragmentation degree, and then the file size and the fragmentation degree can be weighted and summed to obtain the file score, wherein the weight of the file size and the weight of the fragmentation degree can be pre-set by the technician based on experience or experiments, and the file size and the fragmentation degree are both positively correlated with the score. Then, based on the scores of the multiple files, the merging order of the multiple files can be determined according to the rule that the scores are positively correlated with the merging order.
[0066] See also Figure 2 , Figure 2A schematic diagram of a file sorting process provided in an embodiment of the present application, assuming that there are three files: file A, file B, and file C. Assuming that the score of file A is 90, the score of file B is 30, and the score of file C is 10, then the merging order of file A, file B, and file C is: file A, file B, file C. That is, file A and file B are merged first, and then file C is merged.
[0067] In this way, smaller and more fragmented files will be merged first, and larger and more integrated files will be merged later. This conforms to the characteristics of merging from easy to difficult and step by step, which is conducive to maximizing CPU utilization efficiency and reducing overall merging time.
[0068] In step 102, based on the merging order and in accordance with the rule of merging two files each time, multiple files are merged into one file through multiple threads, wherein the two files to be merged each time are merged according to the following steps: based on the current hardware resource information and the file information of the two files, the available buffer size is determined; based on the available buffer size, the file contents of the two files are mapped to the memory, and the file contents in the memory are merged.
[0069] The two files to be merged each time may be files that have not been merged (i.e., initial files), one may be a file that has not been merged and the other may be a merged file, or both may be merged files. The hardware resource information includes a parameter value of at least one hardware parameter, and the file information includes a parameter value of at least one file parameter. Figure 3 The process shown determines the available buffer size and includes the following steps.
[0070] In step 1021, a first weight is determined according to the parameter value of the at least one hardware parameter.
[0071] In some embodiments, there is only one parameter value of a hardware parameter, such as memory usage or CPU load rate. In this case, the product of the parameter value of this hardware parameter and the weight of this hardware parameter can be determined as the first weight, wherein the weight of this hardware parameter can be determined in advance.
[0072] In some embodiments, the parameter value of the at least one hardware parameter includes memory usage and CPU load rate. In this case, the memory usage and CPU load rate can be weighted and summed to obtain a first weight, wherein the weight of the memory usage and the weight of the CPU load rate can be predetermined.
[0073] In step 1022, a second weight is determined according to the parameter value of the at least one file parameter.
[0074] Taking the example that the parameter value of at least one file parameter includes the file size, in the specific implementation, at least two initial files corresponding to the two files to be merged can be determined based on the merge information of the two files to be merged, that is, it can be determined which initial files the two files are merged from, and then based on the file sizes of the at least two initial files, the target file size such as the average file size is determined, and the product of the target file size and the weight of the file size is determined as the second weight, wherein the weight of the file size can be determined in advance.
[0075] Taking the example that the parameter value of at least one file parameter includes a merge order, in a specific implementation, at least two initial files corresponding to the two files to be merged can be determined based on the merge information of the two files, and then the target order can be determined based on the merge order of the at least two initial files, such as determining the product of the target order and the order weight as the second weight, wherein the order weight can be determined in advance.
[0076] Taking the example that the parameter value of at least one file parameter includes file size and merge order, in the specific implementation, at least two initial files corresponding to the two files to be merged can be determined based on the merge information of the two files, and then the target file size can be determined based on the file size of the at least two initial files, and the target order can be determined based on the merge order of the at least two initial files. Then, the second weight is calculated according to the formula: second weight = target file size * file size weight + target order * order weight, wherein the file size weight and the order weight can be determined in advance.
[0077] In step 1023, the difference between the preset value and the weight sum is determined as the buffer weight, and the weight sum refers to the sum of the first weight and the second weight.
[0078] Among them, if the preset value is 1, then the buffer weight=1-(first weight+second weight).
[0079] In step 1024, the product of the preset cache size and the buffer weight is determined as the available cache size.
[0080] The preset cache size may be a maximum buffer size set by the system.
[0081] In practical applications, the available cache size determines the size of the sliding window. For example, the size of the data read from the file each time by the sliding window is set to the available cache size. Figure 4 , Figure 4A schematic diagram of a memory mapping process provided for an embodiment of the present application uses a sliding window to read file content from the starting position of the file, maps the read file content to the cache through memory mapping technology, and then merges the file content in the cache. After that, the sliding window continues to read content from the file, and the memory mapping and merging operations are repeated until the merged content is completed.
[0082] In addition, for each merge, the MD5 values of the two files before the merge can be calculated. After the two files are merged, the MD5 value of the corresponding file content of each file in the merged file can also be calculated. If this MD5 value is different from the MD5 value corresponding to the file before the merge, it means that the merge is wrong and the two files need to be merged again. Only when the MD5 values of the two files are the same as the MD5 values corresponding to the two files before the merge, it means that the merge is correct. And after determining that any merge is correct, the index information can be recorded. The index information includes the corresponding relationship between each merged file and the merged file, and the corresponding line number range of each merged file in the corresponding merged file.
[0083] See also Figure 5 , Figure 5 A schematic diagram of a process for comparing the MD5 values of files before and after merging is provided in an embodiment of the present application. It is assumed that there are three files: file A, file B, and file C that need to be merged, each file contains 100 records, and the merging order is: file A, file B, file C. It is assumed that before the merge, the MD5 value of file A is: XXXA, the MD5 value of file B is: XXXB, and the MD5 value of file C is: XXXC.
[0084] Then, you can first merge file A and file B into file AB. Assuming that lines 0-100 in file AB are the file content of file A, and lines 101-200 are the file content of file B, then you can calculate the MD5 value of the file content of lines 0-100 in file AB: XXXA′, calculate the MD5 value of the file content of lines 101-200 in file AB: XXXB′. If XXXA′ is different from XXXA, or XXXB′ is different from XXXB, it means that the merge is wrong and file A and file B need to be merged again. If XXXA′ is the same as XXXA and XXXB′ is also the same as XXXB, it means that the merge is correct and the MD5 value of file AB can be calculated: XXXA-B.
[0085] In addition, index information may also be saved: file AB contains the contents of file A and file B, and lines 0-100 are the file contents of file A, and lines 101-200 are the file contents of file B.
[0086] After that, merge file AB and file C into file ABC. Lines 0-100 of file ABC are the file contents of file A, lines 101-200 are the file contents of file B, and lines 201-300 are the file contents of file C. Then, the MD5 values of the file contents of lines 0-200 of file ABC can be calculated: XXXA-B′, and the MD5 values of the file contents of lines 201-300 of file ABC can be calculated: XXXC′. If XXXA-B′ is different from XXXA-B, or XXX C′ is different from XXX C, it means that the merge is wrong, and file AB and file C need to be merged again. If XXXA-B′ is the same as XXXA-B, and XXX C′ is the same as XXX C, it means that the merge is correct, and the MD5 value of file ABC can be calculated: XXXA-BC. In this process, the file contents of lines 0-200 of file ABC can be compared as a whole, without having to compare the MD5 values of file A and file B separately, so as to reduce the number of comparisons.
[0087] In addition, index information can also be saved: file ABC contains the contents of file A, file B, and file C, and lines 0-100 are the file contents of file A, lines 101-200 are the file contents of file B, and lines 201-300 are the file contents of file C.
[0088] In this way, verification is performed after each merge, providing an effective real-time monitoring and error correction mechanism, which can effectively reduce the bit error rate and ensure the accuracy and integrity of the merge. In addition, after each merge, only two pairs of MD5 values need to be compared to determine whether the merge is correct. The determination method is relatively simple and has little impact on the merge speed.
[0089] Subsequently, if it is determined that any merged file has been updated, the merged file containing this file and the line range corresponding to this file in the corresponding merged file can be determined based on the index information. Then, the file content corresponding to the line range can be deleted from the determined merged file, and the file can be merged back into the target merged file.
[0090] See also Figure 6 , continue to use Figure 5 In the example, assume that file A, file B, and file C have been merged into file ABC, and lines 0-100 of file ABC are the file content of file A, lines 101-200 are the file content of file B, and lines 201-300 are the file content of file C.
[0091] Assume that file B (containing 100 records) is updated to file B1 (containing 110 records). According to the index information, file ABC contains file C, and lines 201-300 are the file content of file C. Then, the file content of lines 201-300 in file ABC can be deleted, so that file ABC becomes file AC. Lines 0-100 in file AC are the file content of file A, and lines 101-200 are the file content of file C. Afterwards, file AC and file B1 are merged into file AC-B1. Lines 0-100 in file AC-B1 are the file content of file A, lines 101-200 are the file content of file C, and lines 201-310 are the file content of file B1.
[0092] In this way, when a merged file needs to be updated, you only need to find the corresponding merged file, cut and merge the merged file, which is more convenient to operate.
[0093] In addition, multi-threading technology can be used to use a work-stealing algorithm to perform file merging operations in parallel to further speed up the file merging process.
[0094] See also Figure 7 , Assume that there are 5 files to be merged: file A, file B, file C, file D and file E, and the merging order is: file A, file B, file C, file D and file E. Assume that 2 threads are used: thread 1 and thread 2 to merge the files.
[0095] Then, while thread 1 is merging file A and file B, thread 2 can also merge file C and file D. After that, thread 2 can merge file AB and file CD into file ABCD, and then merge file ABCD and file E into file ABCDE.
[0096] The solution of the embodiment of the present application is introduced below with reference to specific examples.
[0097] In actual applications, each time a merge is performed, the available buffer size may be determined based on current hardware resource information and file information of the two files to be merged.
[0098] For example, the available buffer size bufferSize is determined according to the following formula:
[0099] bufferWeight=1-((memoryWeight*memoryUsage)+(cpuWeight*cpuLoad)+(fileWeight*(fileSize / fileSizeConstant))+(priorityWeight*mergePriority)),
[0100] bufferSize=Math.min(maxBufferSize*bufferWeight,maxBufferSize),
[0101] in:
[0102] Memory weight memoryWeight, which indicates the weight of memory usage when calculating buffer weight bufferWeight. The value is usually between 0 and 1. It controls the influence of memory usage on bufferWeight. A higher memoryWeight means that memory usage has a greater influence on buffer weight.
[0103] Memory usage memoryUsage, which is a percentage value between 0 and 1 (for example, 10% usage is represented by 0.1), is used to reflect the current memory resource usage and affects the adjustment of the buffer size;
[0104] cpuWeight, which indicates the weight of CPU load when calculating bufferWeight. Its value is usually between 0 and 1. It is used to control the influence of CPU load on bufferWeight. A higher cpuWeight means that CPU load has a greater influence on buffer weight.
[0105] cpu load cpuLoad, which takes a percentage value between 0 and 1 (for example, 50% load is represented by 0.5), is used to reflect the current CPU resource usage and affects the adjustment of the buffer size;
[0106] File weight fileWeight, which indicates the weight of the file size when calculating bufferWeight. The value is usually between 0 and 1. It is used to control the influence of the file size on bufferWeight. A higher fileWeight means that the file size has a greater influence on the buffer weight.
[0107] The file size fileSize, in MB, is used to calculate bufferWeight after normalization;
[0108] File size constant fileSizeConstant, a constant used to normalize fileSize. By dividing fileSize by fileSizeConstant, the file size can be scaled to a smaller range (usually between 0 and 1) for comparison with other parameters. The choice of fileSizeConstant can be adjusted according to the specific scenario and the range of file sizes;
[0109] Priority weight priorityWeight, which indicates the weight of the merge order when calculating bufferWeight, is usually between 0 and 1. It is used to control the influence of the merge order on bufferWeight. A higher priorityWeight means that the merge priority has a greater influence on the buffer weight.
[0110] Merge priority mergePriority, or merge order (the earlier the merge order, the higher the merge priority), indicates the merge priority of the current file. It is usually an integer, and the range depends on the specific implementation (for example, 1 represents the highest priority, and the larger the value, the lower the priority). It is used to reflect the merge order or importance of files and affects the adjustment of the buffer size.
[0111] Determining the available buffer size this way has the following advantages:
[0112] 1. Flexibility in resource utilization: Considering file size and merging priority helps to dynamically adjust the buffer size to better match the actual merging needs;
[0113] 2. Merge efficiency: By giving priority to files with higher scores, system resources can be released faster and the overall merge efficiency can be improved;
[0114] 3. System stability: Adjusting the buffer size helps maintain system stability under high load and prevents resource overload;
[0115] 4. Load balancing: Taking multiple factors into consideration can allocate resources more evenly and prevent bottlenecks caused by a single factor.
[0116] Further assume that the parameters are as follows:
[0117] maxBufferSize=5000MB, memoryWeight=1, cpuWeight=0.5, fileWeight=0.1, priorityWeight=0.1, fileSizeConstant=10000.
[0118] And suppose there are six files to be merged and their sizes are as follows:
[0119] File A': 10MB,
[0120] File B′: 20MB,
[0121] File C′: 3000MB,
[0122] File D′: 5000MB,
[0123] File E': 6000MB,
[0124] File F′: 10000MB.
[0125] Assume that the merge order of these six files is: A′(1)>B′(2)>C′(3)>D′(4)>E′(5)>F′(6), where the number in brackets indicates the merge order of the files. The larger the number, the later the merge order.
[0126] And assume that the thread pool has 2 threads: thread 1 and thread 2.
[0127] The specific merging process is as follows:
[0128] Phase 1: Thread 1 merges file A′ and file B, 10MB+20MB=30MB.
[0129] Assume that before thread 1 merges file A' and file B', the memory usage is 10%, the CPU load is 10%, and the average size of the two files is: (10+20) / 2=15MB, and the weighted average of the merge order is: (1+2) / 2=1.5, then:
[0130] bufferWeight=1-((memoryWeight*memoryUsage)+(cpuWeight*cpuLoad)+(fileWeight*(fileSize / fileSizeConstant))+(priorityWeight*mergePriority));
[0131] =1-((1*0.1)+(0.5*0.1)+(0.1*15 / 10000)+(0.1*1.5))=0.69985,
[0132] bufferSize=Math.min(5000*0.69985, 5000)=3499.25MB,
[0133] Memory occupied: Math.min(3499.25, 30) = 30MB,
[0134] Number of loops: 1, that is, the sliding window can map the file contents of file A' and file B' to the memory at one time, and merge the file contents in the memory once. Thread 2 waits.
[0135] Phase 2: While thread 1 is processing, thread 2 merges file C′ and file D′, 3000MB+5000MB=8000MB.
[0136] Assume that before thread 2 merges file C' and file D', the memory usage is 12% and the CPU load is 20%, and the average size of the two files is: (3000+5000) / 2=4000MB, and the weighted average of the merge order is: (3+4) / 2=3.5, then:
[0137] bufferWeight=1-((memoryWeight*memoryUsage)+(cpuWeight*cpuLoad)+(fileWeight*(fileSize / fileSizeConstant))+(priorityWeight*mergePriority));
[0138] =1-((1*0.12)+(0.5*0.2)+(0.1*4000 / 10000)+(0.1*3.5))=0.39,
[0139] bufferSize=Math.min(5000*0.39, 5000)=1950MB,
[0140] Memory occupied: Math.min(1950, 8000) = 1950MB;
[0141] Number of cycles: 8000 / 1950>4=5 times. That is, the sliding window needs to map the file contents of file C′ and file D′ to the memory 5 times, and the file contents in the memory also need to be merged 5 times.
[0142] In the third stage, thread 1 merges file A′ and file B′ to obtain file A′-B′, and waits for resources to be released. Thread 2 processes the second cycle of file C′ and file D′.
[0143] In the fourth stage, thread 1 merges file A′-B′ and file E′, 10MB+20MB+6000MB=6030MB.
[0144] Assume that the current memory usage is 15%, the CPU load is 20%, the average size of the files to be merged is: (10+20+6000) / 3=2010MB, and the weighted average of the merge order is: (1+2+5) / 3=2.67, then:
[0145] bufferWeight=1-((memoryWeight*memoryUsage)+(cpuWeight*cpuLoad)+(fileWeight*(fileSize / fileSizeConstant))+(priorityWeight*mergePriority));
[0146] =1-((1*0.15)+(0.5*0.2)+(0.1*2010 / 10000)+(0.1*2.67))=0.4629,
[0147] bufferSize=Math.min(5000*0.4629, 5000)=2314.5MB,
[0148] Memory occupied: Math.min(2314.5, 6030) = 2314.5MB,
[0149] Number of loops: 6030 / 2314.5>2=3 times, that is, the sliding window needs to map the file contents of file A′-B′ and file E′ to the memory 3 times, and the file contents in the memory also need to be merged 3 times.
[0150] Thread 2 processes the last 1950MB of data of file C' and file D'.
[0151] In the fifth stage, thread 1 completes the first cycle of merging file A′-B′ and file E′, processes 2314.5MB of data, and needs to process 3715.5MB; thread 2 completes the task and waits for resource release.
[0152] In the sixth stage, thread 1 completes the task of merging files A′-B′ and E′, obtains file A′-B′-E′, and waits for new tasks.
[0153] Thread 2 merges file C′-D′ and file F′, 3000MB+5000MB+10000MB=18000MB.
[0154] Assume that the memory usage is 15%, the CPU load is 20%, and the average size of the two files is: (3000+5000+10000) / 3=6000MB, and the weighted average of the merge order is: (3+4+6) / 3=4.33, then,
[0155] bufferWeight=1-((memoryWeight*memoryUsage)+(cpuWeight*cpuLoad)+(fileWeight*(fileSize / fileSizeConstant))+(priorityWeight*mergePriority));
[0156] =1-((1*0.15)+(0.5*0.2)+(0.1*6000 / 10000)+(0.1*4.33))=0.257,
[0157] bufferSize=Math.min(5000*0.257, 5000)=1285MB,
[0158] Memory occupied: Math.min(1285, 18000) = 1285MB,
[0159] Number of cycles: 18000 / 1285>14=14 times.
[0160] In the seventh stage, thread 2 completes the task of merging file C′-D′ and file F′, and obtains file C′-D′-F′. Thread 2 merges file A′-B′-E′ and file C′-D′-F′.
[0161] The calculation process is similar to the above process and will not be repeated here.
[0162] Next, the specific process of each merger is introduced below.
[0163] Specific tasks of file merging: merge the contents of multiple files into one file line by line, retaining the original line order.
[0164] Take merging file A′ and file B′ as an example. Assume that the content of file A′ is as follows:
[0165] Line 1: Fund information 1,
[0166] Line 2: Fund information 2,
[0167] Line 3: Fund information 3,
[0168] And assume that the content of file B' is as follows:
[0169] Line 1: Fund information 4,
[0170] Line 2: Fund information 5,
[0171] Line 3: Fund information6.
[0172] Then, the content of the merged file A′-B′ is as follows:
[0173] Line 1: Fund information 1,
[0174] Line 2: Fund information 2,
[0175] Line 3: Fund information 3,
[0176] Line 4: Fund information 4,
[0177] Line 5: Fund information 5,
[0178] Line 6: Fund information6.
[0179] The actual operation steps are as follows:
[0180] 1. Read file contents: Map the file contents of file A′ and file B′ into memory.
[0181] 2. Merge content: Append the content of file A′ to the content of file ′B.
[0182] 3. Write to new file: Write the merged content to a new file or overwrite the original file.
[0183] Finally, the process of merging large files is introduced.
[0184] When processing large files, you usually do not map the entire file into memory at once, but read and write it in batches to avoid out-of-memory problems.
[0185] Assumptions are as follows:
[0186] Available buffer size: 3750MB,
[0187] File C′ size: 3000MB,
[0188] File D′ size: 5000MB,
[0189] Assuming that the small file is read into memory first, the processing flow is as follows:
[0190] First loop (buffer set to 3750MB):
[0191] The first 3000MB of file C′ and the first 750MB of file D′ (3750MB in total) are read into the memory, that is, the memory contains the entire contents of file C′ and a portion of the contents of file D′.
[0192] The data in the memory is merged, for example, the content of file C' is written into a temporary file, and then part of the content of file D' is appended to the temporary file.
[0193] Second loop (buffer set to 3750MB):
[0194] Read the next 3750MB of data from file D' into memory (750MB of file D' has been processed, so the first 3750MB of the remaining 4250MB is read). That is, all the data in memory belongs to file D';
[0195] Append this part of the content of file D′ in memory to the end of the temporary file.
[0196] Third loop (buffer set to 3750MB):
[0197] Read the remaining 500MB of data from file D′ into the memory, that is, the data in the memory still belongs to file D′;
[0198] Append the last 500MB of the contents of the file D′ in memory to the end of the temporary file.
[0199] At this point, the merged temporary file C′-D′ contains all the contents of file C′ and file D′.
[0200] Based on the same technical concept, the embodiment of the present application also provides a file merging device. The principle of solving the problem by the file merging device is similar to that of the above-mentioned file merging method. Therefore, the implementation of the file merging device can refer to the implementation of the file merging method, and the repeated parts will not be repeated.
[0201] Figure 8 A schematic diagram of the structure of a file merging device provided in an embodiment of the present application includes:
[0202] The sorting module 801 is used to merge and sort the acquired multiple files to obtain a merged order of the multiple files;
[0203] A merging module 802, configured to merge the multiple files into one file through multiple threads based on the merging order and according to a rule of merging two files each time;
[0204] The two files to be merged are merged each time according to the following steps:
[0205] The available buffer size is determined according to the current hardware resource information and the file information of the two files, and the file contents of the two files are mapped to the memory according to the available buffer size, and the file contents in the memory are merged.
[0206] In some embodiments, the hardware resource information includes a parameter value of at least one hardware parameter, and the file information includes a parameter value of at least one file parameter. The merging module 802 is specifically configured to:
[0207] Determining a first weight according to a parameter value of the at least one hardware parameter;
[0208] Determining a second weight according to a parameter value of the at least one file parameter;
[0209] Determine the difference between the preset value and the weight sum as the buffer weight, where the weight sum refers to the sum of the first weight and the second weight;
[0210] The product of the preset cache size and the buffer weight is determined as the available cache size.
[0211] In some embodiments, the parameter value of the at least one hardware parameter includes a memory usage rate and a CPU load rate, and the merging module 802 is specifically used to:
[0212] The memory usage rate and the CPU load rate are weighted and summed to obtain the first weight.
[0213] In some embodiments, the parameter value of the at least one file parameter includes file size and merging order, and the merging module 802 is specifically configured to:
[0214] Determine at least two initial files corresponding to the two files according to the merge information of the two files;
[0215] Determine a target file size according to the file sizes of the at least two initial files, and determine a target order according to the merging order of the at least two initial files;
[0216] The target file size and the target order are weighted and summed to obtain the second weight.
[0217] In some embodiments, the sorting module 801 is specifically configured to:
[0218] Performing a weighted summation on the file size and the degree of fragmentation of each file to obtain a score for the file;
[0219] Based on the scores of the multiple files, the merging order of the multiple files is determined according to a rule that the scores are positively correlated with the merging order.
[0220] In some embodiments, a verification module 803 is further included, which is used to:
[0221] After merging the file contents in the memory, the MD5 value of the file contents of each file in the merged file is calculated;
[0222] If the MD5 value corresponding to any file is different from the MD5 value corresponding to the file before the merge, it is determined that the merge is wrong, and the two files are merged again.
[0223] In some embodiments, a recording module 804 is further included, which is used to record index information after each successful merge, wherein the index information includes the corresponding relationship between each merged file and the merged file, and the line number range of each merged file in the corresponding merged file;
[0224] The update module 805 is used to determine, if it is determined that any merged file has been updated, the target merged file containing the file and the line number range corresponding to the file in the target merged file based on the index information; delete the file content corresponding to the line number range from the target merged file, and merge the updated file into the target merged file.
[0225] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated into a processor, or may exist physically alone, or two or more modules may be integrated into one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0226] After introducing the file merging method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.
[0227] Refer to the following Fig. 9 The electronic device 130 implemented according to this embodiment of the present application is described. Fig. 9 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0228] like Fig. 9 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).
[0229] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0230] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .
[0231] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0232] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0233] In an exemplary embodiment, a storage medium is also provided, and when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned file merging method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0234] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor may execute the steps of any file merging method provided in the embodiments of the present application.
[0235] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in the present application.
[0236] Furthermore, the computer program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0237] The program product for file merging in the embodiment of the present application may adopt a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
[0238] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0239] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
[0240] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0241] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.
[0242] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0243] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0244] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0245] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0246] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0247] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0248] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also includes these modifications and variations.
Claims
1. A file merging method, characterized in that: include: Merging and sorting the acquired multiple files to obtain a merged order of the multiple files; Based on the merging order, according to a rule of merging two files at a time, merging the multiple files into one file through multiple threads; The two files to be merged are merged each time according to the following steps: The available buffer size is determined according to the current hardware resource information and the file information of the two files, and the file contents of the two files are mapped to the memory according to the available buffer size, and the file contents in the memory are merged.
2. The method according to claim 1, characterized in that The hardware resource information includes a parameter value of at least one hardware parameter, the file information includes a parameter value of at least one file parameter, and determining the available buffer size according to the current hardware resource information and the file information of the two files includes: Determining a first weight according to a parameter value of the at least one hardware parameter; Determining a second weight according to a parameter value of the at least one file parameter; Determine the difference between the preset value and the weight sum as the buffer weight, where the weight sum refers to the sum of the first weight and the second weight; The product of the preset cache size and the buffer weight is determined as the available cache size.
3. The method according to claim 2, characterized in that The parameter value of the at least one hardware parameter includes a memory usage rate and a CPU load rate, and determining a first weight according to the parameter value of the at least one hardware parameter includes: The memory usage rate and the CPU load rate are weighted and summed to obtain the first weight.
4. The method according to claim 2, characterized in that The parameter value of the at least one file parameter includes a file size and a merge order, and determining a second weight according to the parameter value of the at least one file parameter includes: Determine at least two initial files corresponding to the two files according to the merge information of the two files; Determine a target file size according to the file sizes of the at least two initial files, and determine a target order according to the merging order of the at least two initial files; The target file size and the target order are weighted and summed to obtain the second weight.
5. The method according to claim 1, characterized in that Merging and sorting the acquired multiple files to obtain a merged order of the multiple files includes: Performing a weighted summation on the file size and the degree of fragmentation of each file to obtain a score for the file; Based on the scores of the multiple files, the merging order of the multiple files is determined according to a rule that the scores are positively correlated with the merging order.
6. The method according to claim 1, characterized in that After merging the file contents in memory, it also includes: Calculate the MD5 value of the file content of each file in the merged file; If the MD5 value corresponding to any file is different from the MD5 value corresponding to the file before the merge, it is determined that the merge is wrong, and the two files are merged again.
7. The method according to any one of claims 1 to 6, characterized in that: Also includes: After each successful merge, record index information, the index information including the correspondence between each merged file and the merged file, and the line number range of each merged file in the corresponding merged file; If it is determined that any merged file is updated, determining a target merged file containing the file and a row number range corresponding to the file in the target merged file according to the index information; The file content corresponding to the line number range is deleted from the target merged file, and the updated file is merged into the target merged file.
8. A file merging device, characterized in that: include: A sorting module is used to merge and sort the acquired multiple files to obtain a merged order of the multiple files; A merging module, configured to merge the multiple files into one file through multiple threads based on the merging order and according to a rule of merging two files each time; The two files to be merged are merged each time according to the following steps: The available buffer size is determined according to the current hardware resource information and the file information of the two files, and the file contents of the two files are mapped to the memory according to the available buffer size, and the file contents in the memory are merged.
9. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method as claimed in any one of claims 1 to 7.
11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.
