Data backup method and device, electronic equipment, storage medium and program product
By sorting database backup files and selecting compression algorithms, the problems of low compression efficiency and slow processing speed in traditional database backups are solved, and more efficient data backup and recovery are achieved.
Patent Information
- Application Number
- CN202411961030.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-05-02
AI Technical Summary
In traditional database backup, a single compression algorithm cannot take into account the characteristics of different types of files, resulting in low compression efficiency and slow processing speed.
By classifying backup files, selecting appropriate compression algorithms based on file size and type, compressing and merging large files and small files respectively, reducing metadata storage needs.
Improves compression efficiency and processing speed, reduces metadata storage requirements, and improves overall backup performance.
Smart Images
Figure CN119917473A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a data backup method, device, electronic device, storage medium and program product. Background Art
[0002] During the database backup process, files are compressed using a compression algorithm to reduce the volume of backup data, thereby saving storage space and improving transmission efficiency.
[0003] However, in traditional database backup, most compression tools use a single compression algorithm to compress all files, which often fails to take into account the characteristics of different types of files, resulting in problems such as low compression efficiency and slow processing speed. Summary of the invention
[0004] The embodiments of the present application provide a data backup method, device, electronic device, storage medium and program product to take into account the characteristics of different types of files and improve compression efficiency and processing speed.
[0005] In a first aspect, an embodiment of the present application provides a data backup method, including:
[0006] Scan the original backup directory to obtain multiple backup files, and determine whether the file size of each backup file is greater than a first preset threshold;
[0007] Classify the backup files whose number is greater than the first preset threshold as first-category files, and classify the backup files whose number is less than or equal to the first preset threshold as second-category files;
[0008] Selecting a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files;
[0009] Merging each file in the second category into a target file, and matching a corresponding compression algorithm for the target file;
[0010] Compressing each file in the first category and the target file based on a compression algorithm corresponding to each file;
[0011] Store the compressed file in the target backup directory.
[0012] In a possible implementation manner, selecting a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files includes:
[0013] Get the header data of the preset length of each file in the first category;
[0014] Determine the file type of each file in the first category according to the correspondence between the file type and the header data;
[0015] According to the correspondence between the file type and the compression algorithm, the compression algorithm corresponding to each file in the first category of files is determined.
[0016] In a possible implementation manner, compressing each file in the first category of files and the target file includes:
[0017] When the number of threads in the thread pool is greater than the number of files in the first category of files, compressing each file in the first category of files and the target file in parallel;
[0018] When the number of threads in the thread pool is equal to the number of files in the first category, compressing each file in the first category, and then compressing the target file;
[0019] When the number of threads in the thread pool is less than the number of files in the first category, after compressing some files in the first category, the remaining files in the first category and the target file are compressed in parallel.
[0020] In a possible implementation, compressing each file in the first category of files based on a compression algorithm corresponding to each file includes:
[0021] Determine whether the file size of each file in the first category is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold;
[0022] Classify files larger than the second preset threshold into a first group of files, and classify files smaller than or equal to the second preset threshold into a second group of files;
[0023] Divide each file in the first group of files into blocks so that each block is larger than the first preset threshold and smaller than the second preset threshold;
[0024] Based on a compression algorithm corresponding to each file, the blocks corresponding to the first group of files and the second group of files are compressed.
[0025] In a possible implementation, the method further includes:
[0026] Obtain a compressed file from the target backup directory, and decompress the compressed file.
[0027] In a possible implementation manner, decompressing the compressed file includes:
[0028] Obtaining meta information of the compressed file, the meta information at least including a file path of a backup file corresponding to the compressed file and a compression algorithm corresponding to the compressed file;
[0029] The compressed file is decompressed according to the compression algorithm corresponding to the compressed file, and is decompressed to the file path of the backup file corresponding to the compressed file.
[0030] In a second aspect, the present application provides a data backup device, the device comprising:
[0031] A judgment module, used to scan the original backup directory to obtain multiple backup files, and judge whether the file size of each backup file is greater than a first preset threshold;
[0032] a classification module, configured to classify backup files whose value is greater than the first preset threshold as first-category files, and classify backup files whose value is less than or equal to the first preset threshold as second-category files;
[0033] A selection module, configured to select a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files;
[0034] A merging module, used for merging each file in the second category into a target file, and matching a corresponding compression algorithm for the target file;
[0035] A compression module, used for compressing each file in the first category and the target file based on a compression algorithm corresponding to each file;
[0036] The storage module is used to store the compressed files in the target backup directory.
[0037] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;
[0038] The memory stores computer-executable instructions;
[0039] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.
[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.
[0042] The data backup method, device, electronic device, storage medium and program product provided by the embodiments of the present application, by classifying the backup files, classify the backup files greater than the first preset threshold value into the first category of files, and classify the backup files less than or equal to the first preset threshold value into the second category of files. For the first category of files, the compression algorithm of each file is selected according to the file type of each file in the first category of files, so that the characteristics of different types of files can be taken into account and the compression efficiency and processing speed can be improved. For the second category of files, after each file in the second category of files is merged into a target file, the corresponding compression algorithm is matched for the target file, which can reduce the demand for metadata storage and improve the overall backup performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0044] Figure 1 A schematic diagram of a scenario of the data backup method provided in this application;
[0045] Figure 2 Schematic diagram of the data backup method provided in this application Figure 1 ;
[0046] Figure 3 Schematic diagram of the data backup method provided in this application Figure 2 ;
[0047] Figure 4 Schematic diagram of the data backup method provided in this application Figure 3 ;
[0048] Figure 5 A schematic diagram of the structure of the data backup device provided by this application;
[0049] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.
[0050] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0052] In the process of database backup, compression technology is widely used to reduce storage requirements and speed up backup transmission. Common backup compression methods include ZIP compression algorithm, Gzip compression algorithm, Zstd (Zstandard) compression algorithm, LZ4 compression algorithm, etc. These compression algorithms help reduce the volume of backup data by compressing files during the backup process, thereby saving storage space and improving transmission efficiency.
[0053] However, existing compression methods have some problems:
[0054] 1. In traditional database backup, most compression tools use a single compression algorithm to compress all files. These compression algorithms may produce different effects in compression ratio and processing speed, and often fail to take into account the characteristics of different types of files. Therefore, when facing different types of data, there are problems such as low compression efficiency and slow processing speed.
[0055] 2. Database backups often contain a large number of small files. These small files are usually small in size, such as only a few KB or MB, but they are huge in number and stored in a scattered manner. Since each small file needs to store metadata separately, there is a certain amount of time and space overhead, and the storage and processing efficiency is low, which affects the overall backup performance.
[0056] Therefore, how to improve the compression efficiency during database backup and reduce metadata overhead when processing small files is a relatively important issue.
[0057] To this end, the present application provides a compression method, which is divided into first-class files and second-class files according to file size. For the first-class files, the compression algorithm of each file is selected according to the file type of each file in the first-class files. For the second-class files, each file in the second-class files is merged into a target file, and the corresponding compression algorithm is matched for the target file. Then, based on the compression algorithm corresponding to each file, each file in the first-class files and the target file are compressed. By matching the corresponding compression algorithm for each file in the first-class files, the characteristics of different types of files can be taken into account, and the compression efficiency and processing speed can be improved. By merging each file in the second-class files and then compressing them, the demand for metadata storage can be reduced and the overall backup performance can be improved.
[0058] Figure 1 A schematic diagram of a scenario of the data backup method provided in this application, such as Figure 1 As shown, the electronic device 101 obtains the original backup directory from the original database 102, scans the original backup directory, obtains multiple backup files, determines whether the file size of each backup file is greater than the first preset threshold, and classifies the backup files greater than the first preset threshold as first-category files, and classifies the backup files less than or equal to the first preset threshold as second-category files. Then, the electronic device selects the compression algorithm corresponding to each file in the first-category file according to the file type of each file in the first-category file, merges each file in the second-category file into a target file, matches the corresponding compression algorithm for the target file, and then compresses each file in the first-category file and the target file based on the compression algorithm corresponding to each file. After the electronic device 101 compresses the backup file, it stores the compressed file in the target database 103. The target database 103 and the original database 102 can be the same database or different databases.
[0059] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0060] Figure 2 Schematic diagram of the data backup method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:
[0061] S201. Scan the original backup directory to obtain multiple backup files.
[0062] During the database backup process, the original backup directory is scanned, and the original backup directory includes multiple backup files to be backed up. Therefore, after the original backup directory is scanned, multiple backup files to be backed up can be obtained.
[0063] For example, the original backup directory may be recursively scanned to obtain the meta information of each backup file in the original backup directory. The meta information may include file size, extension, file path, modification time, etc. The modification time may be the time when the file was last modified, and the extension is used to identify the type of the file.
[0064] S202: Determine whether the file size of each backup file is greater than a first preset threshold.
[0065] In this step, it can be determined whether the file size of each backup file is greater than a first preset threshold value according to the file size of each backup file.
[0066] For example, the first preset threshold may be 64 MB, and the first preset threshold may also be determined according to actual conditions.
[0067] S203: Classify backup files whose number is greater than a first preset threshold as first category files.
[0068] In this embodiment, backup files larger than a first preset threshold are classified as first-category files, and files in the first category may also be called large files.
[0069] Then, execute step S205.
[0070] S204: Classify the backup files whose number is less than or equal to the first preset threshold as second-category files.
[0071] In this embodiment, the backup files whose size is smaller than or equal to the first preset threshold are classified as second-category files, and the files in the second-category files may also be called small files.
[0072] For example, a distributed file system stores data in blocks, and files whose size is less than or equal to the block size are called small files, that is, they are classified as second-class files, and files whose size is greater than or equal to the block size are called large files, that is, they are classified as first-class files.
[0073] Then, execute step S206.
[0074] S205: Select a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files.
[0075] In this embodiment, each file in the first category of files is identified, the file type of each file in the first category of files is determined according to the identification result, and the compression algorithm of each file in the first category of files is determined according to the file type of each file in the first category of files.
[0076] For example, for text or log files, due to their high repetitiveness, you can choose a deep compression algorithm, such as the Zstd compression algorithm. For image files, in order to maintain image quality and ensure that the compressed image can fully restore the data of the original image after decompression, you can choose a lossless compression algorithm, such as the Zopfli compression algorithm. Similarly, for binary files, in order to ensure that no data is lost during the compression process, you can choose a lossless compression algorithm, such as the LZ4 compression algorithm, which can achieve faster compression. For structured data, such as database snapshots, JSON, and XML files, you can use a lossless compression algorithm, such as the Zstd compression algorithm. The Zstd compression algorithm can reduce the data size while maintaining the original content of the data, with high compression efficiency and fast speed.
[0077] Then, execute step S207.
[0078] S206: Merge each file in the second category of files into a target file, and match a corresponding compression algorithm for the target file.
[0079] In this embodiment, multiple small files are sequentially concatenated into a large file, and the content of each small file is appended to the target file to generate a complete large file, so as to reduce the proportion of metadata and the frequency of input / output operations.
[0080] For example, you can create a new target file and append the content of each file in the second category to the target file, or you can select a small file and append the content of each file in the second category to the small file.
[0081] Then, a corresponding compression algorithm is matched for the target file. For example, the compression algorithm corresponding to the target file may be a Zstd compression algorithm. If a higher compression ratio is required, an LZMA compression algorithm may be used.
[0082] In practical applications, the corresponding compression algorithm can be matched to the target file according to requirements.
[0083] In some embodiments, during the splicing process, in order to be able to trace the position of each small file (each file in the second type of file) in the large file (target file), the offset of each small file in the large file can be recorded, and the offset indicates the starting position of the small file in the large file. The file path, file size, verification time, modification time and name of the large file of each small file can also be recorded. And the information such as the offset, file path, file size, verification time, modification time and name of the large file of the above-mentioned small files is stored in the backup list file. The backup file Qingdao can be used as an index of the small file, storing the path of each small file and its position in the large file, ensuring that a single small file can be quickly retrieved and restored later.
[0084] Then, execute step S208.
[0085] S207: Compress each file in the first category of files based on the compression algorithm corresponding to each file in the first category of files.
[0086] In this embodiment, each file in the first category of files is compressed based on the compression algorithm corresponding to each file in the first category of files, that is, different compression algorithms are used for different types of files, so as to take into account the characteristics of different types of files, meet the compression requirements of different file types, and improve compression efficiency and processing speed.
[0087] Then, execute step S209.
[0088] S208: Compress the target file based on a compression algorithm corresponding to the target file.
[0089] In this embodiment, the large file formed by merging the small files is compressed using a preset compression algorithm.
[0090] Then, execute step S209.
[0091] In some embodiments, when the number of threads in the thread pool is greater than the number of files in the first category of files, each file in the first category of files and the target file are compressed in parallel; when the number of threads in the thread pool is equal to the number of files in the first category of files, each file in the first category of files is compressed in parallel, and then the target file is compressed; when the number of threads in the thread pool is less than the number of files in the first category of files, after some files in the first category of files are compressed, the remaining files in the first category of files and the target file are compressed in parallel. In this embodiment, during the parallel compression process, the system will start multiple threads (or multiple processes) to process large files and large files of merged small files. Large files and merged files are placed in two different task queues respectively, and the compression process is accelerated in a parallel manner: the system maintains two task queues, one is a "large file compression queue" and the other is a "small file merged large file compression queue". The priority of compressing large files is higher, and resources will be allocated to large file compression tasks first. If there are idle threads in the thread pool, the compression task of large files after merging small files will be started. Each task (whether it is a large file or a merged large file) will be assigned to an independent thread for processing. Each thread uses a corresponding compression algorithm to compress files. Each thread performs file compression operations independently, making full use of the computing power of multi-core processors.
[0092] S209: Store the compressed files in the target backup directory.
[0093] In this embodiment, after compressing the file, the compressed file is stored in the target backup directory, thereby completing the data backup.
[0094] For example, the backup information table may also be updated, and the backup information table records the relevant meta information of each compressed file, such as the file path of the backup file, the compression algorithm, the compressed file size, etc.
[0095] In some embodiments, after the compressed file is stored in the target backup directory, if the backup file needs to be obtained, for example, the data in the database is deleted or lost, or the database server fails, the compressed file can be obtained from the target backup directory and decompressed so that the backup file can be obtained in time when the backup file is needed. The compressed file here can be a compressed file corresponding to any large file, or a compressed file corresponding to the target file.
[0096] For example, the meta information of the compressed file can be obtained, and the meta information at least includes the file path of the backup file corresponding to the compressed file and the compression algorithm corresponding to the compressed file. Accordingly, the compressed file can be decompressed according to the compression algorithm corresponding to the compressed file, and decompressed to the file path of the backup file corresponding to the compressed file, thereby obtaining the backup file, which is convenient for managing and finding the backup file.
[0097] The data backup method provided in the present application matches corresponding compression algorithms for different types of large files, merges multiple small files and matches the corresponding compression algorithms, reduces metadata overhead through a merging strategy and then performs compression, thereby avoiding resource waste caused by the dispersion of small files and algorithm mismatch problems in large file compression.
[0098] Figure 3 Schematic diagram of the data backup method provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, step S205, selecting a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files, is described in detail. The method includes:
[0099] S301, obtaining header data of a preset length for each file in the first category.
[0100] For example, each file in the first category of files is opened, and header data of a preset length of each file is read. The preset length may be, for example, 4 to 16 bytes.
[0101] S302: Determine the file type of each file in the first category of files according to the correspondence between the file type and the header data.
[0102] For example, the header data of each file in the read first category of files is matched with the header data in a predefined file signature database, and then the file type of each file in the first category of files is determined based on the correspondence between the file type and the header data in the file signature database.
[0103] S303: Determine the compression algorithm corresponding to each file in the first category of files according to the correspondence between the file type and the compression algorithm.
[0104] For example, the compression algorithm corresponding to each file in the first category of files is determined according to a predefined correspondence between file types and compression algorithms.
[0105] The data backup method provided in the embodiment of the present application determines the compression algorithm corresponding to each file in the first category of files more quickly and efficiently based on the correspondence between the file type and the header data, and the correspondence between the file type and the compression algorithm.
[0106] Figure 4 Schematic diagram of the data backup method provided in this application Figure 3 ,like Figure 4 As shown, in this embodiment Figure 2 Based on the embodiment, step S207, compressing each file in the first category of files based on the compression algorithm corresponding to each file in the first category of files, is described in detail. The method includes:
[0107] S401: Determine whether each file in the first category of files is larger than a second preset threshold, and whether the second preset threshold is larger than the first preset threshold.
[0108] In this embodiment, it is determined whether each file in the first category is too large, that is, whether each file is larger than a second preset threshold value. The second preset threshold value can be determined according to actual conditions.
[0109] S402: Classify files whose values are greater than a second preset threshold into a first group of files, and classify files whose values are less than or equal to the second preset threshold into a second group of files.
[0110] In this embodiment, files in the first category whose size is greater than the second preset threshold are classified into a first group of files, and files in the first category whose size is less than or equal to the second preset threshold are classified into a second group of files.
[0111] S403: Divide each file in the first group of files into blocks, so that each block is larger than a first preset threshold and smaller than a second preset threshold.
[0112] In this embodiment, each file in the first group of files is processed into blocks, and each block corresponding to each file in the first group of files is larger than a first preset threshold and a second preset threshold.
[0113] S404: Compress the blocks corresponding to the first group of files and the second group of files based on the compression algorithm corresponding to each file.
[0114] For example, each block in the first group of files is compressed based on the compression algorithm of the original file to which it belongs. For example, if a file in the first group of files corresponds to the LZ4 compression algorithm, each block of the file after being divided into blocks may also be compressed based on the LZ4 compression algorithm.
[0115] The data backup method provided in the present application can divide the large file into blocks if the large file is too large, and then compress the blocks to improve the compression efficiency of the large file.
[0116] Figure 5 A schematic diagram of the structure of the data backup device provided in this application, such as Figure 5 As shown, the data backup device 50 provided in this embodiment includes:
[0117] A judgment module 51 is used to scan the original backup directory to obtain multiple backup files, and to judge whether the file size of each backup file is greater than a first preset threshold;
[0118] A classification module 52, configured to classify backup files whose value is greater than the first preset threshold as first-category files, and classify backup files whose value is less than or equal to the first preset threshold as second-category files;
[0119] A selection module 53, configured to select a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files;
[0120] A merging module 54, configured to merge each file in the second category of files into a target file, and match a corresponding compression algorithm for the target file;
[0121] A compression module 55, used for compressing each file in the first category of files and the target file based on a compression algorithm corresponding to each file;
[0122] The storage module 56 is used to store the compressed files in the target backup directory.
[0123] In one possible implementation, the selection module 53 is specifically used to obtain header data of a preset length for each file in the first category of files; determine the file type of each file in the first category of files based on the correspondence between the file type and the header data; and determine the compression algorithm corresponding to each file in the first category of files based on the correspondence between the file type and the compression algorithm.
[0124] In one possible implementation, the compression module 55 is specifically used to compress each file in the first category of files and the target file in parallel when the number of threads in the thread pool is greater than the number of files in the first category of files; to compress the target file after compressing each file in the first category of files when the number of threads in the thread pool is equal to the number of files in the first category of files; and to compress the remaining files in the first category of files and the target file in parallel after compressing some files in the first category of files when the number of threads in the thread pool is less than the number of files in the first category of files.
[0125] In a possible implementation, the compression module 55 is specifically used to determine whether the file size of each file in the first category of files is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold; divide the files greater than the second preset threshold into a first group of files, and divide the files less than or equal to the second preset threshold into a second group of files; divide each file in the first group of files into blocks so that each block is greater than the first preset threshold and less than the second preset threshold; based on the compression algorithm corresponding to each file, compress the blocks corresponding to the first group of files and the second group of files.
[0126] In a possible implementation, a decompression module is further included, and the decompression module is used to obtain a compressed file from the target backup directory and decompress the compressed file.
[0127] In a possible implementation, the decompression module is specifically used to obtain the metadata of the compressed file, the metadata including at least the file path of the backup file corresponding to the compressed file and the compression algorithm corresponding to the compressed file; decompress the compressed file according to the compression algorithm corresponding to the compressed file, and decompress it to the file path of the backup file corresponding to the compressed file.
[0128] The data backup device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.
[0129] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus.
[0130] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.
[0131] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0132] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0133] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.
[0134] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0135] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0136] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0137] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0138] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0139] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0141] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0142] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0143] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0144] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A data backup method, characterized in that: include: Scan the original backup directory to obtain multiple backup files, and determine whether the file size of each backup file is greater than a first preset threshold; Classify the backup files whose number is greater than the first preset threshold as first-category files, and classify the backup files whose number is less than or equal to the first preset threshold as second-category files; Selecting a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files; Merging each file in the second category into a target file, and matching a corresponding compression algorithm for the target file; Compressing each file in the first category and the target file based on a compression algorithm corresponding to each file; Store the compressed file in the target backup directory.
2. The method according to claim 1, characterized in that The selecting, according to the file type of each file in the first category of files, a compression algorithm corresponding to each file in the first category of files comprises: Get the header data of the preset length of each file in the first category; Determine the file type of each file in the first category according to the correspondence between the file type and the header data; According to the correspondence between the file type and the compression algorithm, the compression algorithm corresponding to each file in the first category of files is determined.
3. The method according to claim 1, characterized in that The compressing each file in the first category and the target file comprises: When the number of threads in the thread pool is greater than the number of files in the first category of files, compressing each file in the first category of files and the target file in parallel; When the number of threads in the thread pool is equal to the number of files in the first category, compressing each file in the first category, and then compressing the target file; When the number of threads in the thread pool is less than the number of files in the first category, after compressing some files in the first category, the remaining files in the first category and the target file are compressed in parallel.
4. The method according to claim 1, characterized in that: The compressing each file in the first category of files based on the compression algorithm corresponding to each file includes: Determine whether the file size of each file in the first category is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold; Classify files larger than the second preset threshold into a first group of files, and classify files smaller than or equal to the second preset threshold into a second group of files; Divide each file in the first group of files into blocks so that each block is larger than the first preset threshold and smaller than the second preset threshold; Based on a compression algorithm corresponding to each file, the blocks corresponding to the first group of files and the second group of files are compressed.
5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtain a compressed file from the target backup directory, and decompress the compressed file.
6. The method according to claim 5, characterized in that The decompressing the compressed file comprises: Obtaining meta information of the compressed file, the meta information at least including a file path of a backup file corresponding to the compressed file and a compression algorithm corresponding to the compressed file; The compressed file is decompressed according to the compression algorithm corresponding to the compressed file, and is decompressed to the file path of the backup file corresponding to the compressed file.
7. A data backup device, characterized in that: The device comprises: A judgment module, used to scan the original backup directory to obtain multiple backup files, and judge whether the file size of each backup file is greater than a first preset threshold; a classification module, configured to classify backup files whose value is greater than the first preset threshold as first-category files, and classify backup files whose value is less than or equal to the first preset threshold as second-category files; A selection module, configured to select a compression algorithm corresponding to each file in the first category of files according to the file type of each file in the first category of files; A merging module, used for merging each file in the second category into a target file, and matching a corresponding compression algorithm for the target file; A compression module, used for compressing each file in the first category and the target file based on a compression algorithm corresponding to each file; The storage module is used to store the compressed files in the target backup directory.
8. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.
10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.