File merging method and device, electronic device and storage medium

Abstract

The disclosed embodiment provides a file merging method and device, an electronic device and a storage medium. The file merging method is executed by an electronic device, and the method includes: determining a file merging mode; determining whether there is a merging requirement according to the file merging mode; when there is a merging requirement, merging the target files in the target directory; wherein the data volume of the target files in the target directory is less than the data volume of the non-target files in the target directory.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to a file merging method and device, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of computer technology, the amount of data generated by various systems of electronic equipment has also increased, while the processing of data has become more and more complicated, and the human cost, learning and use cost have also increased.

[0003] When electronic devices are running, even files of the same type will generate a large number of fragmented files, which makes the speed of reading data tasks in electronic devices slower and slower. Electronic devices need to find a large number of files and perform addressing operations, which puts a great pressure on the memory and resources of electronic devices. Manual processing of fragmented files will result in low processing efficiency, high error rate, and increased learning costs. Summary of the invention

[0004] Embodiments of the present disclosure provide a file merging method and device, an electronic device, and a storage medium.

[0005] A first aspect of an embodiment of the present disclosure provides a file merging method, which is executed by an electronic device, and the method includes:

[0006] Determine the file merging mode;

[0007] Determine whether there is a merging requirement according to the file merging mode;

[0008] When there is a merging requirement, the target files under the target directory are merged; wherein the data volume of the target files under the target directory is smaller than the data volume of the non-target files under the target directory.

[0009] Based on the above solution, determining the file merging mode includes:

[0010] The file merging mode is determined according to the load rate of the electronic device, available resource status information, an operation instruction received from a user interface UI, and a pre-configured file merging strategy.

[0011] Based on the above solution, determining whether there is a merging requirement according to the file merging mode includes:

[0012] When the merge mode is the first mode, determining whether there are files in the target directory that are smaller than a merge threshold;

[0013] If there are files in the target directory that are smaller than the merge threshold, it is determined that there is a merge requirement.

[0014] Based on the above solution, determining whether there is a merging requirement according to the file merging mode includes:

[0015] When the merge mode is the second mode, determining whether the average data volume of each file in the target directory is less than a merge threshold;

[0016] When the average data volume is less than the merging threshold, it is determined that there is a need to merge files.

[0017] Based on the above solution, determining whether there is a merging requirement according to the file merging mode includes:

[0018] When the merging mode is the third mode, determining the proportion of the target files in the target directory;

[0019] When the proportion of the target files is greater than the proportion threshold, it is determined that there is a need to merge files.

[0020] Based on the above solution, when there is a merging requirement, merging the target files in the target directory includes:

[0021] Move the target file under the target directory to a first temporary directory;

[0022] Merging the target files in the first temporary directory to obtain a merged file;

[0023] The merged file is written to a second temporary directory.

[0024] Based on the above solution, the method further includes:

[0025] After the merged file is written into the second temporary directory, determining whether the target file is merged successfully;

[0026] When the target files are merged successfully, the merged files in the second temporary directory are moved to the target directory, and the first temporary directory and the second temporary directory are deleted.

[0027] Based on the above solution, the method further includes:

[0028] After deleting the first temporary directory and the second temporary directory, a merge prompt message is generated.

[0029] Based on the above solution, after the merged file is written into the second temporary directory, determining whether the target file is merged successfully includes:

[0030] According to whether the file contents of the target file in the first temporary directory are all contained in the merged file in the second temporary directory;

[0031] When the file contents of the target files in the first temporary directory are all contained in the merged file in the second temporary directory, it is determined that the target files are merged successfully.

[0032] Based on the above solution, the step of determining whether the file content of the target file in the first temporary directory is completely contained in the merged file in the second temporary directory includes:

[0033] Determine respectively a first number of file contents of the target file in the first temporary directory and a second number of file contents in the merged file in the second temporary directory;

[0034] When the second number is not equal to the first number, determining that the target file merge fails;

[0035] When the second number is equal to the first number, the file content of the target file in the first temporary directory and the file content in the merged file are checked, and the target files are determined to be successfully merged based on the checking result.

[0036] Based on the above solution, the method further includes:

[0037] It is determined that the target file merge fails, and the files in the target directory are restored according to the first temporary directory.

[0038] Based on the above solution, determining that the target file merge fails and restoring the files in the target directory according to the first temporary directory includes:

[0039] Record the number of merge failures;

[0040] When the number of merge failures is less than or equal to the retry threshold, merge the target files under the target directory again;

[0041] When the number of merge failures is greater than a retry threshold, the files in the target directory are restored according to the first temporary directory.

[0042] Based on the above solution, the method further includes:

[0043] After restoring the files in the target directory according to the first temporary directory, the first temporary directory and the second temporary directory are deleted.

[0044] Based on the above solution, the method further includes:

[0045] After determining that the target file fails to be merged, the early warning function is triggered to output early warning information of the merger failure.

[0046] Based on the above solution, after deleting the first temporary directory and the second temporary directory, generating merge prompt information includes:

[0047] After deleting the first temporary directory and the second temporary directory, generating an empty file in the target directory;

[0048] According to the prompt that the files are merged successfully, the empty file is named.

[0049] A second aspect of the present disclosure provides a file merging device, which is executed by an electronic device, and includes:

[0050] A first determination module, used to determine a file merging mode;

[0051] A second determination module is used to determine whether there is a merging requirement according to the file merging mode;

[0052] The merging module is used to merge the target files under the target directory when there is a merging demand; wherein the data volume of the target files under the target directory is smaller than the data volume of the non-target files under the target directory.

[0053] Based on the above solution, the first determination module is specifically used to determine the file merging mode according to the load rate of the electronic device, available resource status information, operation instructions received from the user interface UI and the file merging pre-configured strategy.

[0054] Based on the above scheme, the second determination module is specifically used to determine whether there are files smaller than the merge threshold in the target directory when the merge mode is the first mode; if there are files smaller than the merge threshold in the target directory, it is determined that there is a merge requirement.

[0055] Based on the above scheme, the second determination module is specifically used to determine whether the average data volume of each file in the target directory is less than the merge threshold when the merge mode is the second mode; when the average data volume is less than the merge threshold, it is determined that there is a need to merge files.

[0056] Based on the above scheme, the second determination module is specifically used to determine the proportion of the target files under the target directory when the merge mode is the third mode; when the proportion of the target files is greater than the proportion threshold, it is determined that there is a need for file merging.

[0057] Based on the above solution, the merging module includes:

[0058] A moving submodule, used for moving the target file under the target directory to a first temporary directory;

[0059] A merging submodule, used for merging the target files in the first temporary directory to obtain a merged file;

[0060] A writing module is used to write the merged file into a second temporary directory.

[0061] Based on the above solution, the device also includes:

[0062] A third determination module, configured to determine whether the target file is successfully merged after the merged file is written into the second temporary directory;

[0063] A moving module is used to move the merged file in the second temporary directory to the target directory and delete the first temporary directory and the second temporary directory when the target file is merged successfully.

[0064] Based on the above solution, the device also includes:

[0065] A generation module is used to generate merge prompt information after deleting the first temporary directory and the second temporary directory.

[0066] Based on the above solution, the third determination module includes:

[0067] A judging module, configured to determine whether all the file contents of the target file in the first temporary directory are contained in the merged file in the second temporary directory;

[0068] The third determination submodule is used to determine that the target files are successfully merged when the file contents of the target files in the first temporary directory are all contained in the merged file in the second temporary directory.

[0069] Based on the above scheme, the judgment module is specifically used to determine the first number of the file content of the target file in the first temporary directory and the second number of the file content in the merged file in the second temporary directory, respectively; when the second number is not equal to the first number, it is determined that the merge of the target file has failed; when the second number is equal to the first number, the file content of the target file in the first temporary directory and the file content in the merged file are checked, and whether the target files are merged successfully based on the check result.

[0070] Based on the above solution, the device also includes:

[0071] A recovery module is used to determine that the target file merge fails and restore the files in the target directory according to the first temporary directory.

[0072] Based on the above scheme, the recovery module is specifically used to record the number of merge failures; when the number of merge failures is less than or equal to the retry threshold, merge the target files under the target directory again; when the number of merge failures is greater than the retry threshold, restore the files in the target directory according to the first temporary directory.

[0073] Based on the above solution, the device also includes:

[0074] A deletion module is used to delete the first temporary directory and the second temporary directory after restoring the files in the target directory according to the first temporary directory.

[0075] Based on the above solution, the device also includes:

[0076] The alarm module is used to trigger the early warning function after determining that the target file fails to be merged, and output early warning information of the merger failure.

[0077] Based on the above solution, the generation module is specifically used to generate an empty file in the target directory after deleting the first temporary directory and the second temporary directory; and name the empty file according to the prompt of successful file merging.

[0078] A third aspect of the present disclosure provides an electronic device, including:

[0079] a memory for storing processor-executable instructions;

[0080] A processor connected to the memory;

[0081] Wherein, the processor is configured to execute the file merging method provided by any of the aforementioned technical solutions.

[0082] A fourth aspect of an embodiment of the present disclosure provides a non-temporary computer-readable storage medium. When instructions in the storage medium are executed by a processor of a computer, the computer is enabled to execute a file merging method as provided by any of the aforementioned technical solutions.

[0083] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0084] The technical solution provided in the embodiment of the present disclosure, the file merging method includes: determining a file merging mode; determining whether there is a merging demand based on the file merging mode; when there is a merging demand, merging the target files under the target directory; wherein the data volume of the target files under the target directory is less than the data volume of the non-target files under the target directory. The embodiment of the present disclosure provides a variety of merging modes, which facilitates the selection of a merging mode suitable for the current state to merge files according to the actual needs of file merging and / or user wishes. Therefore, the file merging method provided in the embodiment of the present disclosure can realize fast and flexible file merging, improve the efficiency of electronic devices in merging files, and reduce the learning cost of manual merging. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0086] Figure 1 is a flowchart of a file merging method of an electronic device according to an exemplary embodiment;

[0087] Figure 2 is a flowchart of a file merging method of an electronic device according to an exemplary embodiment;

[0088] Figure 3 is a structural schematic diagram of a file merging device according to an exemplary embodiment;

[0089] Figure 4 It is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0090] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0091] like Figure 1 As shown, the present disclosure provides a file merging method, which is performed by an electronic device, and the method includes:

[0092] S110: Determine a file merging mode;

[0093] S120: Determine whether there is a merging requirement according to the file merging mode;

[0094] S130: When there is a merging requirement, merging the target files under the target directory; wherein the data volume of the target files under the target directory is smaller than the data volume of the non-target files under the target directory.

[0095] The electronic device provided in the embodiments of the present disclosure may be various types of electronic devices. For example, the electronic device may be a terminal device or a server. Exemplarily, the terminal device may include: a personal computer (PC), etc. The PC may include a desktop PC, a laptop computer, or a large-screen display device, etc. The server may be: a rack-mounted server and / or a blade server.

[0096] In some embodiments, the electronic device may be an information technology (IT) device. The IT device includes various devices involved in IT computing. For example, the IT device may be a device used as a blockchain node.

[0097] The electronic device may include: a processor and a memory, etc.

[0098] Exemplarily, the processor includes, but is not limited to: a central processing unit, a microcontroller, an embedded controller and / or an image processor.

[0099] Exemplarily, the memory includes, but is not limited to, electronic components such as memory, disk, flash memory or hard disk used for temporary storage and storage of information within the device.

[0100] Exemplarily, the target files in the target directory can be merged by a file merging tool. It can be understood that the file merging tool can be various data processing engines capable of data merging. The data processing engine includes but is not limited to spark.

[0101] It is understandable that the file merging method in the present disclosure can be used for scheduling tasks of a data processing platform. The file merging method can be set to be called periodically. Exemplarily, periodicity means that it is executed at 8 o'clock every day, or on the 1st of each month, etc. The setting of the periodic interval time depends on the specific situation.

[0102] The merge mode refers to the merge threshold, the average data volume of the files in the target directory, or the proportion of files less than the merge threshold. The target file data volume and the number of files determined by the above three methods may be different. Choose different methods to determine the merge mode according to the specific situation. After determining the target file in the target directory, merge the files.

[0103] Since there are many tasks to be performed in an electronic device, multiple tasks often need to be performed at the same time, and the CPU occupancy rate is usually high during the execution of multiple tasks. Therefore, before performing the file merging operation, it is necessary to determine the merging mode for merging files according to the resource usage of the electronic device.

[0104] After determining the file merging mode, it is also necessary to check whether there is a need for merging. For example, if there is no required file in the target directory or the data volume of the file does not meet the preset conditions, there is no need to merge, that is, there is no merge request.

[0105] When the proportion of target files in the target directory or the amount of data in the files meets the preset conditions, there is a need to merge, and the files in the target directory are merged according to the determined merge mode. The target directory refers to the directory of the target files to be merged.

[0106] When merging target files, the required parameters include: storage directory of the merged file, name of the target file, threshold for file merging, first temporary directory, second temporary directory, type of the merged file, and parallelism parameter.

[0107] When the files are written to the target directory, the data volume, number, occupied space, and average data volume of the files written to the target directory are recorded in the data system for easy use when merging files.

[0108] It can be understood that the target files are files of the same type, such as XML data of the same type.

[0109] In some embodiments, the S110 includes:

[0110] The file merging mode is determined according to the load rate of the electronic device, available resource status information, an operation instruction received from a user interface UI, and a pre-configured file merging strategy.

[0111] It can be understood that there are multiple modes for file merging, and different file merging modes can be matched according to instructions of the electronic device or the user.

[0112] It can be understood that the load rate of an electronic device refers to the average number of threads running in a queue in the electronic device within a certain time interval. A high number of threads represents a high load rate, and a low number of threads represents a low load rate.

[0113] The file merging mode is determined according to the load rate threshold. For example, when the load rate of the electronic device is greater than the load rate threshold, the merging mode is set to the second mode or the third mode; when the load rate of the electronic device is less than or equal to the load rate threshold, the merging mode is set to the first mode. By comparing the load rate of the electronic device with the load rate threshold, different merging modes are set. In the process of merging files, the crash of the electronic device due to the load rate is avoided, which leads to the failure of file merging.

[0114] For example, the available resource is the amount of available data in the memory of the electronic device.

[0115] The file merging mode is determined according to the available data volume threshold. For example, when the available data volume of the electronic device is greater than the available data volume threshold, the operating efficiency of the electronic device will be reduced, so the merging mode is set to the second mode or the third mode; when the available data volume of the electronic device is less than or equal to the available data volume threshold, the operating efficiency of the electronic device is higher, so the merging mode is set to the first mode. The merging mode that best suits the electronic device is determined by the available resources, thereby ensuring the stable progress of the file merging. In the process of merging files, the failure of file merging caused by the use of available resources is avoided.

[0116] It is understandable that the operation instructions received from the user interface UI can also affect the file merging mode. For example, the user can set a threshold for file merging or a ratio of files to be merged. When the amount of file data in the target directory is less than the set threshold, the file belongs to the target file. When the number ratio of the target file, that is, the ratio of the total number of files in the target directory exceeds the ratio set by the user, the merging operation is performed. In this way, the mode of merging files is within the scope of user control.

[0117] The user can also set whether to enable merging. For example, if the user sets it to No, the file merging operation will not be performed. If the user sets it to Yes, the merging mode is determined to perform the merging operation. When the user encounters an objective emergency and the user does not need to perform the merging operation, this setting can be used to meet the usage scenario.

[0118] In some embodiments, the S120 includes:

[0119] When the merge mode is the first mode, determining whether there are files in the target directory that are smaller than a merge threshold;

[0120] If there are files in the target directory that are smaller than the merge threshold, it is determined that there is a merge requirement.

[0121] It can be understood that the merge threshold is a parameter set by the user before merging files. For example, the merge threshold of a file can be set to 128M, and then when the data volume of a file is less than 128M, the file is considered to be a target file.

[0122] The first mode is to determine the target file according to the merge threshold, and identify all files smaller than the merge threshold as the target file. Therefore, the first mode can be commonly known as the strict mode. The number of merged files generated after merging using this mode is small, and the merging effect is good. When the CPU occupancy rate of the electronic device is low or the resource usage is good, the first mode can be selected to merge files.

[0123] In some embodiments, the S120 includes:

[0124] When the merge mode is the second mode, determining whether the average data volume of each file in the target directory is less than a merge threshold;

[0125] When the average data volume is less than the merging threshold, it is determined that there is a need to merge files.

[0126] It can be understood that the average data volume in the target directory can also be used to determine the target file. First, the average data volume of the files in the target directory is calculated, and the calculation method is: the total data volume of the target directory / the total number of files in the target directory, so as to obtain the average data volume. When the average data volume of a file is less than the merge threshold preset by the user, the file is the target file.

[0127] The second mode is commonly known as the compromise mode. When the number of files in the target directory is not too large, this mode can be selected to merge files. In this way, when the load rate of the electronic device is high or the available resources are small, the merge mode can be set to the second mode, which can increase the success rate of merging files while ensuring the normal operation of the electronic device.

[0128] In some embodiments, the S120 includes:

[0129] When the merging mode is the third mode, determining the proportion of the target files in the target directory;

[0130] When the proportion of the target files is greater than the proportion threshold, it is determined that there is a need to merge files.

[0131] It is understandable that the third mode needs to obtain a percentage threshold, which is a parameter preset by the user. For example, the percentage threshold is 50%. It is also necessary to obtain the number or data volume of the target files, and the target files are files that are smaller than the preset threshold. When the percentage of the number or data volume of the target files in the current target directory is calculated to be greater than the preset percentage threshold, it is determined that there is a need to merge files. When the percentage of the number or data volume of the target files in the current target directory is less than the preset percentage threshold, there is no need to merge files.

[0132] The third mode is a loose mode, which can be selected to merge files when there are not too many target files in the target directory. In this way, when the load rate of the electronic device is high or the available resources are small, in order to maintain the normal operation of the electronic device, the merging mode can be set to the third mode, which can increase the success rate of merging files while ensuring the normal operation of the electronic device.

[0133] In some embodiments, the S130 includes:

[0134] Move the target file under the target directory to a first temporary directory;

[0135] Merging the target files in the first temporary directory to obtain a merged file;

[0136] The merged file is written to a second temporary directory.

[0137] The first temporary directory and the second temporary directory are directories preset by the user or default directories. The paths and directory names of the first temporary directory, the second temporary directory and the target directory are different.

[0138] The parallelism parameter for merging files is determined based on the files in the first temporary directory. The calculation method is: the total data volume of the files in the first temporary directory is divided by the threshold value to obtain the parallelism parameter. This method can quickly determine the parallelism parameter, so that the target files can be merged according to the parallelism parameter.

[0139] It can be understood that before merging files, the target file is moved to the first temporary directory, which can be used as a directory for backing up the files to be merged, or as a directory for obtaining the files to be merged when merging files. The files backed up in the first temporary directory can be used to restore files when the file merging fails. When restoring files, the files in the first temporary directory can be restored to the target directory.

[0140] The second temporary directory is used to store the merged files. By storing the files before and after merging and the files in the target directory separately, it is possible to ensure that the file merging process can be carried out accurately and improve the efficiency of merging files. And through the files in the first temporary directory, the second temporary directory and the target directory, it is also convenient to view and determine the progress of the file merging.

[0141] In some embodiments, Figure 2 As shown, the method also includes:

[0142] S210: after the merged file is written into the second temporary directory, determining whether the target file is merged successfully;

[0143] S220: When the target files are merged successfully, the merged files in the second temporary directory are moved to the target directory, and the first temporary directory and the second temporary directory are deleted.

[0144] It is understandable that whether the files are merged successfully can be determined by checking the generation of the second temporary directory files, for example, checking the number of files generated and the amount of data.

[0145] In order to save storage space of electronic devices and reduce resource occupancy of electronic devices, after the files are successfully merged, the successfully merged files in the second temporary directory are moved to the target directory. The user can determine whether the current file merging program has ended simply by checking the files in the target directory, which makes it convenient for the user to view the file generation status.

[0146] After the files are successfully merged, the files in the first temporary directory and the second temporary directory are no longer in use, so the first temporary directory and the second temporary directory are deleted, thereby saving the resource occupancy of the electronic device.

[0147] In some embodiments, the method further comprises:

[0148] S310: After deleting the first temporary directory and the second temporary directory, generating merge prompt information.

[0149] It is understandable that after determining that the file merge is successful, a merge prompt message is generated, and the user can further determine that the file merge is successful based on the prompt message. The prompt message facilitates the user to know the status of the current successful merge of the file.

[0150] In some embodiments, the S210 includes:

[0151] S2101: Determine, based on the file contents of the target file in the first temporary directory, whether all of the contents are contained in the merged file in the second temporary directory;

[0152] S2102: When the file contents of the target files in the first temporary directory are all contained in the merged file in the second temporary directory, it is determined that the target files are merged successfully.

[0153] It can be understood that by checking whether the file content is completely contained in the merged file, it can be accurately determined whether the merge is successful.

[0154] It can be understood that when checking whether the file merge is successful, the content of the file, such as information such as keywords, can be checked.

[0155] Exemplarily, the first and last lines of the files may be checked. If the first line is the same as the first line of the first file when merging, and the last line is the same as the last line of the last file when merging, then it may be determined that the file merging is successful; otherwise, the file merging fails.

[0156] Exemplarily, the data volume of the merged file and the data volume of the file before the merge may be checked. If the data volumes of the two files are the same, it may be determined that the file merge is successful. Otherwise, the file merge fails.

[0157] In some embodiments, the S2101 includes:

[0158] Determine respectively a first number of file contents of the target file in the first temporary directory and a second number of file contents in the merged file in the second temporary directory;

[0159] When the second number is not equal to the first number, determining that the target file merge fails;

[0160] When the second number is equal to the first number, the file content of the target file in the first temporary directory and the file content in the merged file are checked, and the target files are determined to be successfully merged based on the checking result.

[0161] It can be understood that the first number refers to the number of file contents in the first temporary directory, and the second number refers to the number of file contents of the merged files in the second temporary directory. If the two numbers are equal, the file contents in the first temporary directory can be compared to see if they are the same, so as to further determine whether the file merge is successful.

[0162] In some embodiments, the method further comprises:

[0163] S410: Determine that the target file merge fails, and restore the files in the target directory according to the first temporary directory.

[0164] Understandably, when the file merge fails, in order not to affect the files in the target directory or to facilitate the subsequent restart of the merge program, the files backed up in the first temporary directory need to be restored to the target directory so that the number and content of files in the target directory are the same as before the merge. Through the restore operation, the merge program can be restarted quickly and smoothly, reducing the possibility of merge errors.

[0165] In some embodiments, the S410 includes:

[0166] Record the number of merge failures;

[0167] When the number of merge failures is less than or equal to the retry threshold, merge the target files under the target directory again;

[0168] When the number of merge failures is greater than a retry threshold, the files in the target directory are restored according to the first temporary directory.

[0169] Understandably, failures may occur during the file merging process due to various reasons, such as resources, storage, or CPU. If manual processing is required every time, it will cost a lot of manpower. By setting the retry threshold, set the retry time period, such as retrying after 5 minutes. The specific time interval depends on the specific situation.

[0170] After a file merge fails, the recorded number of failures can be compared with the retry threshold. If the retry threshold is not exceeded, the files in the target directory can be merged again. This can reduce the consumption of manpower costs and save the learning cost of manpower to restart the merge program.

[0171] When the number of failures exceeds the retry threshold, a recovery operation is performed, that is, the files backed up in the first temporary directory are restored to the target directory.

[0172] In some embodiments, the method further comprises:

[0173] S510: After restoring the files in the target directory according to the first temporary directory, the first temporary directory and the second temporary directory are deleted.

[0174] It can be understood that in the event of a merge failure, after restoring the files in the target directory, since the first temporary directory and the second temporary directory occupy a certain amount of space on the disk, and the files in the first temporary directory and the second temporary directory serve as intermediate directories in the merging process, after the files are merged successfully, the first temporary directory and the second temporary directory will be deleted, thereby saving the occupancy rate of electronic device resources.

[0175] In some embodiments, the method further comprises:

[0176] S610: After determining that the target file fails to be merged, triggering an early warning function, outputting early warning information of the merger failure.

[0177] It is understandable that after a file merge fails, the user can quickly obtain merge failure information through the early warning function, without having to manually check the file information on the electronic device to obtain the file failure, thus improving the efficiency of post-failure file processing.

[0178] Exemplarily, the early warning function includes: sending text messages to specific people. Key mobile phone numbers can be pre-configured to allow users to know more quickly through text messages.

[0179] In some embodiments, the S310 includes:

[0180] After deleting the first temporary directory and the second temporary directory, generating an empty file in the target directory;

[0181] According to the prompt that the files are merged successfully, the empty file is named.

[0182] It can be understood that after the file is merged successfully, an empty file is generated in the target directory, and the empty file is used to indicate that the file may have been successfully ended.

[0183] By way of example, an empty file may be named _SUCCESS.

[0184] In one embodiment, the subsequent strict mode is the first mode, the compromise mode is the second mode, the loose mode is the third mode, and the number of retries is the retry threshold. The spark file merging tool can be used to handle file problems that may occur during big data processing. Based on the information of data collected in the electronic device, files are automatically merged.

[0185] When data is written to disk, relevant important metrics are recorded, including the number of records and files in the partition / table. After the data is written to disk, fragmented file detection is automatically triggered to determine whether the data merge task needs to be started. The service that implements merge files is used to merge table / partition fragmented files, which is called directly by the system or the user.

[0186] The electronic device can automatically detect files and merge them automatically without human intervention. After the data task is completed, the data table will count the number of files in the target directory, the amount of data in each file, and the average amount of data in the file, and record this information in the metadata system.

[0187] Merging files requires a certain amount of execution time and resources (CPU, memory, and network). Users can choose the merge mode to merge files. The merge modes include:

[0188] Strict mode: All files with a size smaller than the threshold must be merged. With this mode, at most only one file after the final merge will have a data size smaller than the set threshold. However, this mode consumes more resources and takes more time.

[0189] Compromise mode: files need to be merged when the average data volume is less than the threshold. This merge mode ensures that the number of files is not less than the number of files without the target file, and the number of files is not too large.

[0190] Loose mode: The number of files smaller than the threshold accounts for a greater percentage of the total number of files than the percentage threshold (for example, the percentage threshold can be 40%, 45%, 50%, 55% or 60%). This merging mode ensures that the number of files is small.

[0191] User-defined: Users can define the proportion of target files (files to be merged) and whether to enable merging.

[0192] According to the merge mode selected by the user, all files with data volume less than the set threshold are moved to the first temporary directory. The total file data volume of these temporary directories is counted and divided by the threshold. The number of final merged files can be calculated. This is used as the parallelism parameter of Spark, and Spark is used to merge files and write the merged results to the second temporary directory.

[0193] After the merge is complete, verify whether the merge is successful by verifying the files in the first temporary directory and the second temporary directory.

[0194] If the merge is successful, the files in the second temporary directory are moved to the original target directory. Finally, the first and second temporary directories are cleaned up and deleted, and an empty file named _SUCCESS is created in the original target directory, indicating that the file merge is complete.

[0195] If the merge fails, the retry operation is performed according to the retry threshold parameter configured by the user. If it still fails, the target directory needs to be restored, the files moved to the first temporary directory are moved back to the target directory, the first temporary directory and the second temporary directory are deleted, and an alarm is sent to the user.

[0196] The file merge is made into a service and integrated into the scheduling system. When user table / partition data is written, the service call is automatically triggered, and the target file is detected and merged. Users can also manually call the service to merge historical directory files.

[0197] like Figure 3 As shown, an embodiment of the present disclosure provides a file merging device, which is executed by an electronic device, and the device includes:

[0198] A first determination module 110, configured to determine a file merging mode;

[0199] A second determination module 120, configured to determine whether there is a merging requirement according to the file merging mode;

[0200] The merging module 130 is used to merge the target files under the target directory when there is a merging demand; wherein the data volume of the target files under the target directory is smaller than the data volume of the non-target files under the target directory.

[0201] In some embodiments, the first determination module 110 is specifically used to determine the file merging mode according to the load rate of the electronic device, available resource status information, operation instructions received from the user interface UI, and a pre-configured file merging strategy.

[0202] In some embodiments, the second determination module 120 is specifically used to determine whether there are files smaller than a merge threshold in the target directory when the merge mode is the first mode; if there are files smaller than the merge threshold in the target directory, determine that there is a merge requirement.

[0203] In some embodiments, the second determination module 120 is specifically used to determine whether the average data volume of each file in the target directory is less than the merge threshold when the merge mode is the second mode; when the average data volume is less than the merge threshold, determine that there is a need to merge files.

[0204] In some embodiments, the second determination module 120 is specifically used to determine the proportion of the target files in the target directory when the merge mode is the third mode; when the proportion of the target files is greater than a proportion threshold, determine that there is a need for file merging.

[0205] In some embodiments, the merging module 130 includes:

[0206] A moving submodule, used for moving the target file under the target directory to a first temporary directory;

[0207] A merging submodule, used for merging the target files in the first temporary directory to obtain a merged file;

[0208] A writing module is used to write the merged file into a second temporary directory.

[0209] In some embodiments, the apparatus further comprises:

[0210] A third determination module, configured to determine whether the target file is successfully merged after the merged file is written into the second temporary directory;

[0211] A moving module is used to move the merged file in the second temporary directory to the target directory and delete the first temporary directory and the second temporary directory when the target file is merged successfully.

[0212] In some embodiments, the apparatus further comprises:

[0213] A generation module is used to generate merge prompt information after deleting the first temporary directory and the second temporary directory.

[0214] In some embodiments, the third determination module includes:

[0215] A judging module, configured to determine whether all the file contents of the target file in the first temporary directory are contained in the merged file in the second temporary directory;

[0216] The third determination submodule is used to determine that the target files are successfully merged when the file contents of the target files in the first temporary directory are all contained in the merged file in the second temporary directory.

[0217] In some embodiments, the judgment module is specifically used to determine the first number of the file content of the target file in the first temporary directory and the second number of the file content in the merged file in the second temporary directory, respectively; when the second number is not equal to the first number, it is determined that the merge of the target file has failed; when the second number is equal to the first number, the file content of the target file in the first temporary directory and the file content in the merged file are checked, and whether the target files are merged successfully based on the check result.

[0218] In some embodiments, the apparatus further comprises:

[0219] A recovery module is used to determine that the target file merge fails and restore the files in the target directory according to the first temporary directory.

[0220] In some embodiments, the recovery module is specifically used to record the number of merge failures; when the number of merge failures is less than or equal to a retry threshold, merge the target files under the target directory again; when the number of merge failures is greater than the retry threshold, restore the files in the target directory based on the first temporary directory.

[0221] In some embodiments, the apparatus further comprises:

[0222] A deletion module is used to delete the first temporary directory and the second temporary directory after restoring the files in the target directory according to the first temporary directory.

[0223] In some embodiments, the apparatus further comprises:

[0224] The alarm module is used to trigger the early warning function after determining that the target file fails to be merged, and output early warning information of the merger failure.

[0225] In some embodiments, the generation module is specifically used to generate an empty file in the target directory after deleting the first temporary directory and the second temporary directory; and name the empty file according to the prompt of successful file merging.

[0226] An embodiment of the present disclosure provides an electronic device, including:

[0227] a memory for storing processor-executable instructions;

[0228] A processor connected to the memory;

[0229] Wherein, the processor is configured to execute the file merging method provided by any of the aforementioned technical solutions.

[0230] The processor may include various types of storage media, which are non-transitory computer storage media that can continue to remember information stored thereon after the electronic device loses power.

[0231] Here, the electronic device includes: a mobile terminal or a server. The electronic device includes a processor and a memory. The processor and the memory are located together in a housing of the electronic device.

[0232] The processor can be connected to the memory through a bus or the like, and is used to read the executable program stored in the memory, for example, to execute the following Figure 1 and Figure 2 At least one of any of the methods shown.

[0233] An embodiment of the present disclosure shows a structure of an electronic device. Figure 4 The electronic device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions executable by the processing component 922, such as an application. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform any of the aforementioned methods applied in the base station, such as Figure 1 and Figure 2 At least one of the methods shown.

[0234] The electronic device 900 may also include a power supply component 926 configured to perform power management of the electronic device 900, a wired or wireless network interface 950 configured to connect the electronic device 900 to a network, and an input / output (I / O) interface 958. The electronic device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0235] The present disclosure provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer, the computer can execute the following Figure 1 and Figure 2 At least one of the file merging methods shown.

[0236] In one embodiment, the file merging method may include: determining a file merging mode; determining whether there is a merging requirement based on the file merging mode; when there is a merging requirement, merging target files under a target directory; wherein the data volume of the target files under the target directory is less than the data volume of non-target files under the target directory.

[0237] It can be understood that the determining of the file merging mode includes: determining the file merging mode according to the load rate of the electronic device, available resource status information, an operation instruction received from a user interface UI, and a file merging pre-configured strategy.

[0238] It can be understood that determining whether there is a merge need based on the file merge mode includes: when the merge mode is the first mode, determining whether there are files smaller than the merge threshold in the target directory; if there are files smaller than the merge threshold in the target directory, determining that there is a merge need.

[0239] It can be understood that determining whether there is a need for merging according to the file merging mode includes: when the merging mode is the second mode, determining whether the average data volume of each file in the target directory is less than the merging threshold; when the average data volume is less than the merging threshold, determining that there is a need for file merging.

[0240] It can be understood that determining whether there is a merging requirement based on the file merging mode includes: when the merging mode is the third mode, determining the proportion of the target files under the target directory; when the proportion of the target files is greater than the proportion threshold, determining that there is a file merging requirement.

[0241] It can be understood that when there is a merging requirement, merging the target files under the target directory includes: moving the target files under the target directory to a first temporary directory; merging the target files under the first temporary directory to obtain a merged file; and writing the merged file to a second temporary directory.

[0242] It can be understood that the method also includes: after the merged file is written to the second temporary directory, determining whether the target file is merged successfully; when the target file is merged successfully, moving the merged file in the second temporary directory to the target directory, and deleting the first temporary directory and the second temporary directory.

[0243] It can be understood that the method further includes: generating merge prompt information after deleting the first temporary directory and the second temporary directory.

[0244] It can be understood that after the merged file is written to the second temporary directory, determining whether the target file is merged successfully includes: determining whether the file content of the target file in the first temporary directory is completely contained in the merged file in the second temporary directory; when the file content of the target file in the first temporary directory is completely contained in the merged file in the second temporary directory, determining that the target file is merged successfully.

[0245] It can be understood that the process of determining whether the file content of the target file in the first temporary directory is entirely contained in the merged file in the second temporary directory includes: respectively determining a first number of the file content of the target file in the first temporary directory and a second number of the file content in the merged file in the second temporary directory; when the second number is not equal to the first number, determining that the merge of the target file has failed; when the second number is equal to the first number, comparing the file content of the target file in the first temporary directory and the file content in the merged file, and determining whether the target files have been merged successfully based on the comparison result.

[0246] It can be understood that the method further includes: determining that the target file merge fails, and restoring the files in the target directory according to the first temporary directory.

[0247] It can be understood that the determination of the target file merging failure and the restoration of the files in the target directory according to the first temporary directory include: recording the number of merge failures; when the number of merge failures is less than or equal to a retry threshold, merging the target files under the target directory again; when the number of merge failures is greater than the retry threshold, restoring the files in the target directory according to the first temporary directory.

[0248] It can be understood that the method further includes: after restoring the files in the target directory according to the first temporary directory, deleting the first temporary directory and the second temporary directory.

[0249] It can be understood that the method further includes: after determining that the target file fails to be merged, triggering an early warning function and outputting early warning information of the merger failure.

[0250] It can be understood that after deleting the first temporary directory and the second temporary directory, generating merge prompt information includes: after deleting the first temporary directory and the second temporary directory, generating an empty file in the target directory; naming the empty file according to the prompt of successful file merger.

[0251] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0252] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A file merging method, characterized in that: Executed by an electronic device, the method includes: Determine a file merging mode according to the load rate of the electronic device, available resource status information, an operation instruction received from a user interface UI, and a pre-configured file merging strategy; the file merging mode includes one or more of a first mode, a second mode, and a third mode; Determine whether there is a need for merging according to the file merging mode; determine whether there is a need for merging according to the file merging mode, including: determine whether there is a need for merging according to the conditions corresponding to the file merging mode; the conditions corresponding to the first mode are: whether there is a target file in the target directory; the conditions corresponding to the second mode are: whether the average data volume of each file in the target directory is less than the merging threshold; the conditions corresponding to the third mode are: whether the proportion of target files in the target directory is greater than the proportion threshold; the target file is a file with a data volume less than the merging threshold; When there is a merging requirement, merge the target files in the target directory.

2. The method according to claim 1, characterized in that The determining whether there is a merging requirement according to the file merging mode includes: When the merge mode is the first mode, determining whether there are files in the target directory that are smaller than a merge threshold; If there are files in the target directory that are smaller than the merge threshold, it is determined that there is a merge requirement.

3. The method according to claim 1 or 2, characterized in that: The determining whether there is a merging requirement according to the file merging mode includes: When the merge mode is the second mode, determining whether the average data volume of each file in the target directory is less than a merge threshold; When the average data volume is less than the merging threshold, it is determined that there is a need to merge files.

4. The method according to claim 1, characterized in that: The determining whether there is a merging requirement according to the file merging mode includes: When the merging mode is the third mode, determining the proportion of the target files in the target directory; When the proportion of the target files is greater than the proportion threshold, it is determined that there is a need to merge files.

5. The method according to claim 1, characterized in that When there is a merging requirement, the target files in the target directory are merged, including: Move the target file under the target directory to a first temporary directory; Merging the target files in the first temporary directory to obtain a merged file; The merged file is written to a second temporary directory.

6. The method according to claim 5, characterized in that The method further comprises: After the merged file is written into the second temporary directory, determining whether the target file is merged successfully; When the target files are merged successfully, the merged files in the second temporary directory are moved to the target directory, and the first temporary directory and the second temporary directory are deleted.

7. The method according to claim 6, characterized in that The method further comprises: After deleting the first temporary directory and the second temporary directory, a merge prompt message is generated.

8. The method according to claim 6, characterized in that After the merged file is written into the second temporary directory, determining whether the target file is merged successfully includes: According to whether the file contents of the target file in the first temporary directory are all contained in the merged file in the second temporary directory; When the file contents of the target files in the first temporary directory are all contained in the merged file in the second temporary directory, it is determined that the target files are merged successfully.

9. The method according to claim 8, characterized in that The determining whether all the file contents of the target file in the first temporary directory are contained in the merged file in the second temporary directory comprises: Determine respectively a first number of file contents of the target file in the first temporary directory and a second number of file contents in the merged file in the second temporary directory; When the second number is not equal to the first number, determining that the target file merge fails; When the second number is equal to the first number, the file content of the target file in the first temporary directory and the file content in the merged file are checked, and the target files are determined to be successfully merged based on the checking result.

10. The method according to claim 9, characterized in that The method further comprises: It is determined that the target file merge fails, and the files in the target directory are restored according to the first temporary directory.

11. The method according to claim 10, characterized in that The determining that the target file merge fails, and restoring the file in the target directory according to the first temporary directory, comprises: Record the number of merge failures; When the number of merge failures is less than or equal to the retry threshold, merge the target files under the target directory again; When the number of merge failures is greater than a retry threshold, the files in the target directory are restored according to the first temporary directory.

12. The method according to claim 10 or 11, characterized in that: The method further comprises: After restoring the files in the target directory according to the first temporary directory, the first temporary directory and the second temporary directory are deleted.

13. The method according to claim 10, characterized in that The method further comprises: After determining that the target file fails to be merged, the early warning function is triggered to output early warning information of the merger failure.

14. The method according to claim 7, characterized in that After deleting the first temporary directory and the second temporary directory, generating merge prompt information includes: After deleting the first temporary directory and the second temporary directory, generating an empty file in the target directory; According to the prompt that the files are merged successfully, the empty file is named.

15. A file merging device, characterized in that: The method is performed by an electronic device, and the device includes: The first determination module is used to determine the file merging mode according to the load rate of the electronic device, available resource status information, operation instructions received from the user interface UI and the pre-configured file merging strategy; the file merging mode includes one or more of the first mode, the second mode and the third mode; the second determination module is used to determine whether there is a merging demand according to the file merging mode; determining whether there is a merging demand according to the file merging mode includes: determining whether there is a merging demand according to the conditions corresponding to the file merging mode; the conditions corresponding to the first mode are: whether there is a target file in the target directory; the conditions corresponding to the second mode are: whether the average data volume of each file in the target directory is less than the merging threshold; the conditions corresponding to the third mode are: whether the proportion of target files in the target directory is greater than the proportion threshold; the target file is a file with a data volume less than the merging threshold; The merge module is used to merge the target files in the target directory when there is a merge requirement.

16. The device according to claim 15, characterized in that The second determination module is specifically used to determine whether there are files smaller than a merge threshold in the target directory when the merge mode is the first mode; if there are files smaller than the merge threshold in the target directory, determine that there is a merge requirement.

17. The device according to claim 15, characterized in that The second determination module is specifically used to determine whether the average data volume of each file in the target directory is less than the merge threshold when the merge mode is the second mode; when the average data volume is less than the merge threshold, determine that there is a need to merge files.

18. The device according to claim 15, characterized in that The second determination module is specifically used to determine the proportion of the target files in the target directory when the merge mode is the third mode; when the proportion of the target files is greater than a proportion threshold, determine that there is a need for file merging.

19. The device according to claim 15, characterized in that The merging module comprises: A moving submodule, used for moving the target file under the target directory to a first temporary directory; A merging submodule, used for merging the target files in the first temporary directory to obtain a merged file; A writing module is used to write the merged file into a second temporary directory.

20. The device according to claim 19, characterized in that The device also includes: A third determination module, configured to determine whether the target file is successfully merged after the merged file is written into the second temporary directory; A moving module is used to move the merged file in the second temporary directory to the target directory and delete the first temporary directory and the second temporary directory when the target file is merged successfully.

21. The device according to claim 20, characterized in that The device also includes: A generation module is used to generate merge prompt information after deleting the first temporary directory and the second temporary directory.

22. The device according to claim 20, characterized in that The third determining module comprises: A judging module, configured to determine whether all the file contents of the target file in the first temporary directory are contained in the merged file in the second temporary directory; The third determination submodule is used to determine that the target files are successfully merged when the file contents of the target files in the first temporary directory are all contained in the merged file in the second temporary directory.

23. The device according to claim 22, characterized in that The judgment module is specifically used to determine the first number of the file content of the target file in the first temporary directory and the second number of the file content in the merged file in the second temporary directory respectively; when the second number is not equal to the first number, determine that the merge of the target file fails; when the second number is equal to the first number, check the file content of the target file in the first temporary directory and the file content in the merged file, and determine whether the target file is merged successfully based on the check result.

24. The device according to claim 23, characterized in that The device also includes: A recovery module is used to determine that the target file merge fails and restore the files in the target directory according to the first temporary directory.

25. The device according to claim 24, characterized in that The recovery module is specifically used to record the number of merge failures; when the number of merge failures is less than or equal to a retry threshold, merge the target files under the target directory again; when the number of merge failures is greater than the retry threshold, restore the files in the target directory according to the first temporary directory.

26. The device according to claim 24 or 25, characterized in that The device also includes: A deletion module is used to delete the first temporary directory and the second temporary directory after restoring the files in the target directory according to the first temporary directory.

27. The device according to claim 24, characterized in that The device also includes: The alarm module is used to trigger the early warning function after determining that the target file fails to be merged, and output early warning information of the merger failure.

28. The device according to claim 21, characterized in that The generation module is specifically used to generate an empty file in the target directory after deleting the first temporary directory and the second temporary directory; and name the empty file according to the prompt of successful file merging.

29. An electronic device, characterized in that: include: a memory for storing processor-executable instructions; A processor connected to the memory; The processor is configured to execute the file merging method provided in any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a computer, enables the computer to execute the file merging method provided in any one of claims 1 to 14.

Images