Data deduplication method and device based on sub-file and readable medium

Abstract

The application discloses a data deduplication method and device based on sub-files and readable medium, obtains an input file, judges whether the input file is an Office file or a Pdf file, in response to determining that the input file is an Office file, circularly analyzes local file data in the Office file, calculates fingerprint information of the local file data, and stores the fingerprint information into a recipe until a central directory record is detected, in response to determining that the input file is a Pdf file, analyzes the Pdf file to obtain stream data, calculates fingerprint information of the stream data, and stores the fingerprint information into the recipe, defines the local file data in the Office file and the stream data in the Pdf file as data blocks, queries whether the fingerprint information corresponding to the data blocks exists in a metadata storage structure array of a persistent storage medium, obtains a query result, and performs deduplication according to the query result, so that the deduplication efficiency can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer storage, and more specifically to a method, apparatus, and readable medium for data deduplication based on subfiles. Background Technology

[0002] With the explosive growth of data, data reduction technologies have emerged. These technologies reduce the amount of data physically stored on storage media by minimizing data redundancy, allowing data centers to process the same amount of data with fewer or less resources, thus reducing data center management costs. Current data reduction technologies include data compression and deduplication. Data compression uses lossless compression algorithms to encode data blocks so that they can be represented with fewer bits. Deduplication prevents the writing of data blocks if identical data blocks already exist in the storage system (i.e., blocks containing exactly the same data).

[0003] To achieve high data reduction rates, current deduplication methods integrate these two technologies and perform lossless compression on deduplicated blocks. Deduplication can reduce data storage costs by effectively eliminating redundant data on a large scale. For example, an Office and PDF file backup server with a daily workload of approximately 50GB would need to back up 18,250GB of data annually. In fact, there is significant redundancy between different versions of a file, as files are typically only added or deleted a small number of times. Traditional backup servers cannot recognize redundancy between files and can only back up large amounts of data. Furthermore, if a backup system employs disaster recovery design, large amounts of data will be transferred to other backup servers to withstand unpredictable errors. This will exponentially increase the cost of purchasing storage equipment for data storage departments. With the development of deduplication technology, it also faces many challenges. Traditional Content Defined Chunking (CDC) algorithms treat files as binary streams, while files have their own structure. For example, productivity files like Office documents and PDF files. An Office document is essentially a zip file with many small sub-file metadata. A PDF file is composed of tags. The data portion enclosed by "stream" and "endstream" is called the stream data, which constitutes the main body of the PDF file. Traditional CDC methods might split tags into two different data blocks, or split a stream of data into two different data blocks, resulting in semantically incorrect chunking. Summary of the Invention

[0004] In view of the aforementioned technical problems, the purpose of the embodiments of this application is to provide a data deduplication method, apparatus, and readable medium based on sub-files to solve the technical problems mentioned in the background section above.

[0005] In a first aspect, the present invention provides a data deduplication method based on subfiles, comprising the following steps:

[0006] Obtain the input file and determine whether it is an Office file or a PDF file;

[0007] In response to determining that the input file is an Office file, the system parses the local file data in the Office file in a loop, calculates fingerprint information for the local file data, and stores it in the recipe until the central directory record is detected.

[0008] In response to the determination that the input file is a PDF file, the PDF file is parsed to obtain stream data, fingerprint information is calculated from the stream data, and stored in the recipe;

[0009] Define local file data in Office files and stream data in PDF files as data blocks, query whether the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, obtain the query results, and perform deleting based on the query results.

[0010] As a preferred option, it also includes:

[0011] Initialize the metadata storage structure array of the persistent storage medium. The metadata in the metadata storage structure array includes fingerprint information, container number, offset of data block in container, and length of data block;

[0012] Initialize an open container, which is a container that is not full and can be used for writing cache. If the open container is full, it becomes a closed container, which is a container that is full and written to disk. Then, a new open container is created.

[0013] Initialize the recipe, which is used to store fingerprint information of data blocks and fingerprint information of metadata blocks.

[0014] Preferably, the input file is determined to be an Office file or a PDF file, specifically including:

[0015] The format of the input file is determined by the magic number of the input file format, thus identifying whether the input file is an Office file or a PDF file.

[0016] As a preferred method, cyclically parsing local file data within an Office file specifically includes:

[0017] Parse the first partial file header in an Office file, obtain the partial file data corresponding to the first partial file header and its compressed length, and calculate the position of the next partial file header based on the length;

[0018] Iterate through and parse the next local file header, obtain the local file data corresponding to the next local file header, until the central directory record is detected.

[0019] Preferably, the PDF file is parsed to obtain stream data, specifically including:

[0020] Detect each stream start flag and its adjacent stream end flag in the PDF file, and treat the data between the stream start flag and its adjacent stream end flag, excluding the barrier byte, as stream data.

[0021] As a preferred option, it also includes:

[0022] Define the remaining parts of the Office file (excluding local file data) and the remaining parts of the PDF file (excluding stream data) as metadata blocks, calculate fingerprint information for the metadata blocks, and store it in the recipe;

[0023] The fingerprint information corresponding to the metadata block does not need to be checked to see if it exists in the metadata storage structure array of the persistent storage medium.

[0024] Preferably, the system queries whether the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, obtains the query results, and performs deduplication based on the query results, specifically including:

[0025] If the fingerprint information corresponding to the data block is found to exist in the metadata storage structure array of the persistent storage medium, then the data block is determined to be duplicate data and there is no need to write the data block to the open container.

[0026] In response to the determination that the fingerprint information corresponding to the data block does not exist in the metadata storage structure array of the persistent storage medium, the data block is determined to be unique data, and the database is written to the open container.

[0027] Secondly, the present invention provides a data deduplication device based on sub-files, comprising:

[0028] The file format determination module is configured to obtain the input file and determine whether the input file is an Office file or a PDF file.

[0029] The first fingerprint calculation module is configured to, in response to determining that the input file is an Office file, cyclically parse the local file data in the Office file, calculate fingerprint information on the local file data, and store it in the recipe, until the central directory record is detected;

[0030] The second fingerprint calculation module is configured to, in response to determining that the input file is a PDF file, parse the PDF file to obtain stream data, calculate fingerprint information from the stream data, and store it in the recipe;

[0031] The query module is configured to define local file data in Office files and stream data in PDF files as data blocks, query whether the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, obtain the query results, and perform deduplication based on the query results.

[0032] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0033] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) The data deduplication method based on sub-files proposed in this invention perceives the structure of the file, and each cut point has a certain semantic meaning, which greatly improves the deduplication efficiency.

[0036] (2) The data deduplication method based on subfiles proposed in this invention only calculates fingerprint information for the metadata blocks of files appearing in Office files and PDF files, but does not perform fingerprint search, thus saving CPU resources. For data blocks, not only is fingerprint information calculated, but fingerprint search is also performed to determine whether the data block is duplicate data, so as to determine whether to perform deduplication and improve the deduplication rate.

[0037] (3) The data deduplication method based on subfiles proposed in this invention intelligently finds the boundaries of subfiles by reusing the metadata blocks inside the file. Compared with the traditional CDC method that requires byte-by-byte calculation, it greatly improves the block division efficiency. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an exemplary device architecture diagram in which an embodiment of this application can be applied;

[0040] Figure 2 This is a flowchart illustrating a sub-file-based data deduplication method according to an embodiment of this application.

[0041] Figure 3 This is a schematic diagram of an Office file structure illustrating a sub-file-based data deduplication method according to an embodiment of this application.

[0042] Figure 4 This is a schematic diagram illustrating the deleting of Office files in a sub-file-based data deleting method according to an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of PDF file structure deleting in the sub-file-based data deleting method of this application;

[0044] Figure 6 A schematic diagram of the PDF file structure for the sub-file-based data deduplication method in an embodiment of this application. Figure 1 ;

[0045] Figure 7 A schematic diagram of the PDF file structure for the sub-file-based data deduplication method in an embodiment of this application. Figure 2 ;

[0046] Figure 8 This is a schematic diagram of a sub-file-based data deduplication apparatus according to an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the present application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0049] Figure 1An exemplary device architecture 100 is shown that can be applied to the sub-file-based data deduplication method or sub-file-based data deduplication apparatus of the embodiments of this application.

[0050] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0051] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, and 103.

[0052] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0053] Server 105 can be a server that provides various services, such as a background data processing server that processes files or data uploaded by terminal devices 101, 102, and 103. The background data processing server can process the acquired files or data and generate processing results.

[0054] It should be noted that the data deduplication method based on subfiles provided in this application embodiment can be executed by server 105 or by terminal devices 101, 102, and 103. Correspondingly, the data deduplication device based on subfiles can be set in server 105 or in terminal devices 101, 102, and 103.

[0055] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. If the data being processed does not need to be retrieved remotely, the above architecture may not include a network, requiring only servers or terminal devices.

[0056] Figure 2An embodiment of this application illustrates a subfile-based data deduplication method, comprising the following steps:

[0057] S1, obtain the input file and determine whether the input file is an Office file or a PDF file.

[0058] In specific embodiments, it also includes:

[0059] S0 initializes the metadata storage structure array of the persistent storage medium. The metadata storage structure array includes fingerprint information, container number, offset of data block in container, and length of data block.

[0060] Initialize an open container, which is a container that is not full and can be used for writing cache. If the open container is full, it becomes a closed container, which is a container that is full and written to disk. Then, a new open container is created.

[0061] Initialize the recipe, which is used to store fingerprint information of data blocks and fingerprint information of metadata blocks.

[0062] Specifically, the initialization steps include:

[0063] (1) Initialize the metadata storage structure array of the deduplication system inside the SSD. A piece of metadata includes: fingerprint information, container number, offset of the data block within the container, and length of the data block. Fingerprint query is to check whether the fingerprint information corresponding to the data block is in the metadata storage structure array. If it exists, it means that the data block is duplicate data; if it does not exist, it means that the data block is unique data.

[0064] (2) Initialize an open container. There are two types of containers: open containers and closed containers. An open container is a container that has not yet been filled and is used as a write cache. A closed container is a container that has been filled and written to the disk. The container size is set to 4MB. Once it is full, the current open container will be closed and become a closed container, and a new open container will be created.

[0065] (3) Initialize the recipe, which is used to store the fingerprint information of the data blocks and the fingerprint information of the metadata blocks of the file.

[0066] In a specific embodiment, step S1, determining whether the input file is an Office file or a PDF file, specifically includes:

[0067] The format of the input file is determined by the magic number of the input file format, thus identifying whether the input file is an Office file or a PDF file.

[0068] Specifically, if the first two characters of the magic number of the input file format are "PK", proceed to step S2; if the first four characters of the magic number of the input file format are "%PDF", proceed to step S3.

[0069] S2, in response to determining that the input file is an Office file, parses the local file data in the Office file in a loop, calculates fingerprint information for the local file data and stores it in the recipe, until the central directory record is detected.

[0070] In a specific embodiment, the process of cyclically parsing local file data within an Office file includes:

[0071] Parse the first partial file header in an Office file, obtain the partial file data corresponding to the first partial file header and its compressed length, and calculate the position of the next partial file header based on the length;

[0072] Iterate through and parse the next local file header, obtain the local file data corresponding to the next local file header, until the central directory record is detected.

[0073] Specifically, the format of Office files is as follows: Figure 3 As shown, the Local file header (LFH) is followed by the local file data. Each local file header and the following local file data constitute a sub-file. The fingerprint information calculated in the embodiments of this application is a hash value, which is obtained by using a hash algorithm. In other embodiments, other fingerprint calculation algorithms may also be used to calculate the fingerprint information.

[0074] Parse the first local file header to obtain the compressed length of the local file data, and calculate the position of the next local file header based on the length. Calculate the hash value of the current local file header and store it in the recipe. The local file header is treated as a metadata block. The hash value calculated from the metadata block does not undergo fingerprint lookup; the current local file header is directly treated as unique data, thus saving CPU resources. Calculate the hash value of the current local file data and store it in the recipe. The local file data is treated as a data block. The hash value calculated from the data block needs to undergo fingerprint lookup. Iterate through the local file headers and local file data in the Office file until the first central directory record is detected. Calculate the hash value of the remaining part of the Office file and store it in the recipe. The remaining part of the Office file is treated as a metadata block. The hash value calculated from the metadata block does not undergo fingerprint lookup; it is directly treated as unique data, thus speeding up the processing. For unique data, metadata needs to be generated, specifically including fingerprint information, container number, data block offset within the container, and data block length. This metadata is stored in the metadata storage structure array initialized in step S0. The specific situation of Office file deduplication is as follows: Figure 4 As shown. If the current data block or metadata block is unique, the data block needs to be written to the current open container. If the current open container is full, the current open container needs to be converted into a closed container and a new open container needs to be reinitialized. If the current data block is duplicate data, it does not need to be written to the open container.

[0075] Specifically, an Office file is essentially a zip file. The file consists of a header and some local data, while the rest of the file comprises central directory records, ending with a central directory end marker. The specific structure is as follows: Figure 4 As shown. The local file header contains a lot of metadata, including the length of the current sub-file. In this embodiment, the length of the current sub-file is obtained by parsing the local file header. Then, each local file header is parsed sequentially to obtain all the local file data. For the local file data, a hash value needs to be calculated. For the local file header, the central directory record, and the final central directory end marker, no fingerprint lookup is needed after calculating the hash value; they are all treated as unique data because the metadata within these files occupies a very small space, and not deleting this part of the data will only cause a small loss in the deduplication rate.

[0076] S3, in response to determining that the input file is a PDF file, parse the PDF file to obtain stream data, calculate fingerprint information from the stream data, and store it in the recipe.

[0077] In a specific embodiment, parsing a PDF file to obtain stream data specifically includes:

[0078] Detect each stream start flag and its adjacent stream end flag in the PDF file, and treat the data between the stream start flag and its adjacent stream end flag, excluding the barrier byte, as stream data.

[0079] In specific embodiments, it also includes:

[0080] Define the remaining parts of the Office file (excluding local file data) and the remaining parts of the PDF file (excluding stream data) as metadata blocks, calculate fingerprint information for the metadata blocks, and store it in the recipe;

[0081] The fingerprint information corresponding to the metadata block does not need to be checked to see if it exists in the metadata storage structure array of the persistent storage medium.

[0082] Specifically, the PDF file has clear boundaries to distinguish between sub-files. In the embodiments of this application, the data between the strings "stream" and "endstream" is treated as a data block, generally referred to as stream data. It should be noted that the barrier bytes are not included in the data block, but rather in the metadata block. See the details below. Figure 5 The PDF file is scanned using the strings "stream" and "endstream" as the start and end markers of the stream, respectively. The search continues until the string "stream" is found. The hash value between the search start point and the string "stream" is calculated and stored in the recipe. The scan continues until the string "endstream" is found. The hash value of the data between "stream" and "endstream" is calculated and stored in the recipe. Each stream start marker has an adjacent stream end marker, and several stream data entries exist within a PDF file. Stream data is the main part of a PDF file. In this embodiment, the hash value is calculated for the stream data. The remaining part of the PDF file is metadata blocks. Although the hash value of the metadata blocks is also calculated, it is not used for fingerprint lookup, saving processor resources. Similarly, the hash values ​​calculated for the data blocks and metadata blocks are stored in the recipe. Specifically, refer to... Figure 6 and Figure 7 Because some special bytes, known as barrier bytes, appear near the stream start and end markers, these barrier bytes can severely impact deduplication efficiency. Embodiments of this application skip these barrier bytes to ensure an objective deduplication rate. These special barrier bytes are generally generated due to updates to a PDF file.

[0083] S4 defines local file data in Office files and stream data in PDF files as data blocks, queries whether the fingerprint information corresponding to the data blocks exists in the metadata storage structure array of persistent storage media, obtains the query results, and performs deleting based on the query results.

[0084] In a specific embodiment, the process involves querying whether the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, obtaining the query result, and performing deduplication based on the query result. Specifically, this includes:

[0085] If the fingerprint information corresponding to the data block is found to exist in the metadata storage structure array of the persistent storage medium, then the data block is determined to be duplicate data and there is no need to write the data block to the open container.

[0086] In response to the determination that the fingerprint information corresponding to the data block does not exist in the metadata storage structure array of the persistent storage medium, the data block is determined to be unique data, and the database is written to the open container.

[0087] Specifically, the hash values ​​calculated for data blocks are fingerprinted. If a hash value exists in the metadata storage structure array, it is considered duplicate data; otherwise, the data block corresponding to that hash value is unique. Hash values ​​calculated from metadata are not fingerprinted and are always treated as unique. Therefore, by reusing the file's internal metadata and intelligently finding the boundaries of subfiles, the block segmentation efficiency is greatly improved compared to the traditional CDC method, which requires byte-by-byte calculation.

[0088] File recovery requires a recipe, which stores fingerprint information of the file's internal metadata blocks and data blocks. During file recovery, the recipe is read to extract the fingerprint information, and then the fingerprint is used for deduplication. The physical address of the corresponding metadata block or data block is then retrieved from the system metadata storage structure array. This physical address is:

[0089] 1. Which container holds the data block or metadata block;

[0090] 2. The offset of the block within the container;

[0091] 3. The length of the block.

[0092] Then, retrieve the current data block or metadata block from the container, repeatedly read the block until all fingerprint information in the recipe has been read, and finally restore an Office file or PDF file.

[0093] The steps S1-S4 above do not represent the order of the steps, but are merely symbolic representations of the steps.

[0094] Further reference Figure 8 As an implementation of the methods shown in the above figures, this application provides an embodiment of a data deduplication device based on subfiles, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0095] This application provides a data deduplication device based on subfiles, including:

[0096] File format determination module 1 is configured to obtain the input file and determine whether the input file is an Office file or a PDF file;

[0097] The first fingerprint calculation module 2 is configured to, in response to determining that the input file is an Office file, cyclically parse the local file data in the Office file, calculate fingerprint information on the local file data, and store it in the recipe, until the central directory record is detected;

[0098] The second fingerprint calculation module 3 is configured to, in response to determining that the input file is a PDF file, parse the PDF file to obtain stream data, calculate fingerprint information from the stream data, and store it in the recipe;

[0099] Query module 4 is configured to define local file data in Office files and stream data in PDF files as data blocks, query whether the fingerprint information corresponding to the data blocks exists in the metadata storage structure array of the persistent storage medium, obtain the query results, and perform deleting based on the query results.

[0100] The following is for reference. Figure 9 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., Figure 1 The diagram shows the structure of a computer device 900 (a server or terminal device). Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0101] like Figure 9As shown, the computer device 900 includes a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 903 or programs loaded from storage section 909 into random access memory (RAM) 904. The RAM 904 also stores various programs and data required for the operation of the device 900. The CPU 901, GPU 902, ROM 903, and RAM 904 are interconnected via a bus 905. An input / output (I / O) interface 906 is also connected to the bus 905.

[0102] The following components are connected to I / O interface 906: an input section 907 including a keyboard, mouse, etc.; an output section 908 including an LCD, speakers, etc.; a storage section 909 including a hard disk, etc.; and a communication section 910 including a network interface card, such as a LAN card or modem. The communication section 910 performs communication processing via a network such as the Internet. A drive 911 may also be connected to I / O interface 906 as needed. A removable medium 912, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 911 as needed so that computer programs read from it can be installed into storage section 909 as required.

[0103] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 910, and / or installed from removable medium 912. When the computer program is executed by central processing unit (CPU) 901 and graphics processing unit (GPU) 902, the functions defined in the methods of this application are performed.

[0104] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0105] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based means to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of this application can be implemented in software or hardware. These modules can also be located within a processor.

[0108] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire an input file and determine whether the input file is an Office file or a PDF file; in response to determining that the input file is an Office file, iteratively parse local file data in the Office file, calculate fingerprint information for the local file data, and store it in the recipe, until a central directory record is detected; in response to determining that the input file is a PDF file, parse the PDF file to obtain stream data, calculate fingerprint information for the stream data, and store it in the recipe; define the local file data in the Office file and the stream data in the PDF file as data blocks, query whether the fingerprint information corresponding to the data blocks exists in the metadata storage structure array of the persistent storage medium, obtain the query result, and perform deduplication based on the query result.

[0109] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data deduplication method based on subfiles, characterized in that, Includes the following steps: Obtain the input file and determine whether the input file is an Office file or a PDF file; In response to determining that the input file is an Office file, the system iteratively parses partial file data within the Office file, specifically including: Parse the first partial file header of the Office file, obtain the partial file data corresponding to the first partial file header and its compressed length, and calculate the position of the next partial file header based on the length; Iterate through and parse the next local file header to obtain the local file data corresponding to the next local file header until the central directory record is detected. Calculate fingerprint information for the local file data and store it in the recipe until the central directory record is detected. In response to determining that the input file is a PDF file, the PDF file is parsed to obtain stream data, specifically including: Detect each stream start flag and its adjacent stream end flag in the PDF file, and take the data between the stream start flag and its adjacent stream end flag, excluding the barrier byte, as stream data. Calculate fingerprint information for the stream data and store it in the recipe. Define the local file data in the Office file and the stream data in the PDF file as data blocks, query whether the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, obtain the query result, and perform deleting based on the query result; The remaining parts of the Office file excluding the local file data and the remaining parts of the PDF file excluding the stream data are defined as metadata blocks. Fingerprint information is calculated for the metadata blocks and stored in the recipe. The fingerprint information corresponding to the metadata block does not need to be queried to see if it exists in the metadata storage structure array of the persistent storage medium.

2. The data deduplication method based on sub-files according to claim 1, characterized in that, Also includes: Initialize the metadata storage structure array of the persistent storage medium. The metadata in the metadata storage structure array includes fingerprint information, container number, offset of data block in container, and length of data block. Initialize an open container, which is a container that is not full and can be used for writing cache. If the open container is full, it becomes a closed container, which is a container that is full and written to disk, and then a new open container is created. Initialize the recipe, which is used to store fingerprint information of data blocks and fingerprint information of metadata blocks.

3. The data deduplication method based on sub-files according to claim 1, characterized in that, The determination of whether the input file is an Office file or a PDF file specifically includes: The format of the input file is determined by the magic number of the input file's format, thus identifying the input file as an Office file or a PDF file.

4. The data deduplication method based on sub-files according to claim 1, characterized in that, The process of querying whether the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, obtaining the query result, and performing deduplication based on the query result specifically includes: In response to determining that the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, the data block is determined to be duplicate data and there is no need to write the data block into the open container; In response to determining that the fingerprint information corresponding to the data block does not exist in the metadata storage structure array of the persistent storage medium, the data block is determined to be unique data, and the data block is written into the open container.

5. A data deduplication device based on sub-files, characterized in that, include: The file format determination module is configured to acquire an input file and determine whether the input file is an Office file or a PDF file. The first fingerprint calculation module is configured to, in response to determining that the input file is an Office file, cyclically parse partial file data within the Office file, specifically including: Parse the first partial file header of the Office file, obtain the partial file data corresponding to the first partial file header and its compressed length, and calculate the position of the next partial file header based on the length; Iterate through and parse the next local file header to obtain the local file data corresponding to the next local file header until the central directory record is detected. Calculate fingerprint information for the local file data and store it in the recipe until the central directory record is detected. The second fingerprint calculation module is configured to parse the PDF file to obtain stream data in response to determining that the input file is a PDF file, specifically including: Detect each stream start flag and its adjacent stream end flag in the PDF file, and take the data between the stream start flag and its adjacent stream end flag, excluding the barrier byte, as stream data. Calculate fingerprint information for the stream data and store it in the recipe. The query module is configured to define the local file data in the Office file and the stream data in the PDF file as data blocks, query whether the fingerprint information corresponding to the data block exists in the metadata storage structure array of the persistent storage medium, obtain the query result, and perform deleting based on the query result; The remaining parts of the Office file excluding the local file data and the remaining parts of the PDF file excluding the stream data are defined as metadata blocks. Fingerprint information is calculated for the metadata blocks and stored in the recipe. The fingerprint information corresponding to the metadata block does not need to be queried to see if it exists in the metadata storage structure array of the persistent storage medium.

6. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

Images