Data deduplication method, device, equipment and medium

Abstract

The present disclosure relates to a data deduplication method, device, equipment and medium. The data deduplication method comprises: obtaining to-be-deduplicated data and identification information corresponding to the to-be-deduplicated data, the identification information comprising a first fingerprint key corresponding to the to-be-deduplicated data; determining a target storage area corresponding to the to-be-deduplicated data based on the identification information, the target storage area corresponding to a preset instance, the preset instance being used to perform a data write operation and / or a data read operation on the target storage area; performing a data read operation based on the preset instance to obtain target storage data in the target storage area, and determining whether the to-be-deduplicated data exists based on the target storage data. According to the embodiments of the present disclosure, the efficiency and accuracy of data deduplication can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data deduplication method, apparatus, device, and medium. Background Technology

[0002] In today's information-explosive era, collecting and storing massive amounts of data has become an important task. However, because massive amounts of data often contain a large amount of duplicate data, the process of collecting and storing massive amounts of data not only wastes storage space but also increases the complexity of data processing and analysis. Therefore, deduplication of massive amounts of data is crucial to improving the efficiency of storing and processing massive amounts of data.

[0003] Existing methods for deduplicating massive amounts of data mainly involve using hash algorithms to calculate the unique identifier of the massive data, obtaining the hash value corresponding to the massive data, and then using Bloom filter technology based on the hash value to deduplicatize the massive data.

[0004] However, existing methods for deduplication of massive amounts of data, which are based on hash algorithms, face the problem of hash collisions when processing massive amounts of data. At the same time, the Bloom filter technology itself has a certain false recognition rate, which leads to the technical problems of low efficiency and accuracy in deduplication of massive amounts of data. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a data deduplication method, apparatus, device, and medium.

[0006] A first aspect of this disclosure provides a data deduplication method, including:

[0007] Obtain the weight data to be de-weighted and the corresponding identification information, including the first fingerprint key corresponding to the weight data to be de-weighted;

[0008] The target storage area corresponding to the data to be deduplicated is determined based on the identification information. The target storage area has a corresponding preset instance, which is used to perform data write operations and / or data read operations on the target storage area.

[0009] The data reading operation is performed based on the preset instance to obtain the target storage data in the target storage area, and the existence of the data to be deduplicated is determined based on the target storage data.

[0010] A second aspect of this disclosure provides a data deduplication device, comprising:

[0011] The data acquisition module is used to acquire the weight data to be deweighted and the corresponding identification information of the weight data to be deweighted. The identification information includes the first fingerprint key corresponding to the weight data to be deweighted.

[0012] The region determination module is used to determine the target storage region corresponding to the deduplication data based on the identification information. The target storage region has a corresponding preset instance, which is used to perform data writing and / or data reading operations on the target storage region.

[0013] The deduplication judgment module is used to perform data reading operations based on a preset instance to obtain the target storage data in the target storage area, and determine whether the data to be deduplicated exists based on the target storage data.

[0014] A third aspect of this disclosure provides an electronic device, including:

[0015] processor;

[0016] Memory, used to store executable instructions;

[0017] The processor is used to read executable instructions from memory and execute the executable instructions to implement the data deduplication method provided in the first aspect above.

[0018] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the data deduplication method provided in the first aspect.

[0019] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0020] The data deduplication method, apparatus, device, and medium provided in this disclosure can acquire deduplicated data and corresponding identification information. The identification information includes a first fingerprint key corresponding to the deduplicated data. Based on the identification information, a target storage area corresponding to the deduplicated data is determined. The target storage area corresponds to a preset instance, which is used to perform data write operations and / or data read operations on the target storage area. Then, based on the preset instance, a data read operation is performed to obtain the target storage data in the target storage area. Based on the target storage data, it is determined whether the deduplicated data exists. Thus, the target storage area corresponding to the deduplicated data can be determined through the identification information of the deduplicated data, and the existence of the deduplicated data can be determined based on the target storage data in the target storage area, thereby improving the efficiency and accuracy of data deduplication. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a data deduplication method provided in an embodiment of this disclosure;

[0024] Figure 2 This is a flowchart of another data deduplication method provided in this embodiment of the disclosure;

[0025] Figure 3 This is a schematic diagram of the structure of a data deduplication device provided in an embodiment of this disclosure;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0028] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0029] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] Typically, existing methods for deduplicating massive amounts of data primarily involve using hash algorithms to calculate the unique identifiers of the data, obtaining hash values, and then applying Bloom filters to deduplicate the data based on these hash values. However, existing methods for deduplicating massive amounts of data using hash algorithms inherently face hash collision problems. Furthermore, Bloom filters themselves have a certain false positive rate, resulting in low efficiency and accuracy in deduplication. To address this issue, this disclosure provides a data deduplication method, which is described below with reference to specific embodiments.

[0033] Figure 1 This is a flowchart of a data deduplication method provided in an embodiment of the present disclosure. The method can be executed by a data deduplication device, which can be implemented in software and / or hardware. The data deduplication device can be configured in an electronic device, such as a server or terminal, wherein the terminal specifically includes a mobile phone, computer or tablet computer, etc.

[0034] like Figure 1 As shown, the data deduplication method provided in this embodiment includes the following steps:

[0035] S110. Obtain the weight data to be de-weighted and the corresponding identification information, including the first fingerprint key corresponding to the weight data to be de-weighted.

[0036] In this embodiment of the disclosure, when the electronic device receives an instruction to deduplicat data, it acquires the data to be deduplicated and the corresponding identification information.

[0037] The data to be deduplicated can be any data that needs to be deduplicated, or data from any database; there are no restrictions here.

[0038] Identification information is used to identify data to be deduplicated.

[0039] In some embodiments of this disclosure, the first fingerprint key corresponding to the deduplicated data can be the identification information inherent in the data to be excluded. After obtaining the deduplicated data, the first fingerprint key corresponding to the deduplicated data can be obtained by parsing the data.

[0040] In some other embodiments of this disclosure, when the first fingerprint key corresponding to the deduplicated data does not exist in the deduplicated data, the first fingerprint key corresponding to the deduplicated data can be calculated and obtained based on a preset hash algorithm. The specific implementation of obtaining the first fingerprint key corresponding to the deduplicated data based on the preset hash algorithm is similar to the existing implementation of fingerprint keys calculated by hash algorithm, and will not be described in detail here.

[0041] S120. Determine the target storage area corresponding to the deduplicated data based on the identification information. The target storage area has a preset instance, which is used to perform data write operations and / or data read operations on the target storage area.

[0042] In this embodiment of the disclosure, after the electronic device obtains the data to be de-weighed and the corresponding identification information, it determines the target storage area corresponding to the data to be de-weighed based on the identification information.

[0043] In this embodiment of the disclosure, the electronic device is provided with multiple storage areas for storing acquired data, including data to be deduplicated. Each storage area can store a preset number of data, which can be pre-set as needed.

[0044] Optionally, the target storage area can be a LevelDB database target storage area. LevelDB is a persistent and efficient key-value database. Based on LevelDB, the persistent preservation and reliability of data can be ensured. Even if the system fails or restarts, the deduplicated data can still be reliably preserved without losing the deduplication results, thus ensuring the sustainability and stability of the data.

[0045] Specifically, after acquiring the data to be deduplicated and the corresponding identification information, the electronic device determines the target storage area corresponding to the data to be deduplicated from multiple storage areas based on the identification information. The storage area corresponding to the identification information of each data to be deduplicated is unique and unchanging. That is to say, for the same data to be deduplicated, when determining its corresponding target storage area based on the identification information of the data to be deduplicated, the corresponding target storage area is the same at all times, ensuring the accuracy of subsequent deduplication of the data to be deduplicated.

[0046] A preset instance is a pre-configured and stored instance used to perform data write and / or data read operations on a target storage area.

[0047] In some embodiments of this disclosure, when the target storage area corresponding to the data to be deduplicated does not exist, the electronic device triggers a storage area creation operation and creates the target storage area corresponding to the identification information and its corresponding preset instance based on the storage area creation operation.

[0048] S130. Perform a data reading operation based on a preset instance to obtain the target storage data in the target storage area, and determine whether the data to be deduplicated exists based on the target storage data.

[0049] In this embodiment of the disclosure, after determining the target storage area corresponding to the data to be deduplicated, the electronic device performs a data reading operation based on a preset instance corresponding to the target storage area to obtain the target storage data in the target storage area, and determines whether the data to be deduplicated exists based on the target storage data.

[0050] Specifically, after determining the target storage area corresponding to the data to be deduplicated, the electronic device performs a data reading operation on the target storage area based on the preset instance corresponding to the target storage area to obtain the target storage data in the target storage area. The target storage data and the data to be deduplicated are compared to determine whether the data to be deduplicated already exists in the target storage area.

[0051] In some embodiments of this disclosure, when the target storage area is created by an electronic device based on a storage area creation operation, it is directly determined that the deduplicated data does not exist.

[0052] In this embodiment, it is possible to obtain the data to be deduplicated and the corresponding identification information. The identification information includes a first fingerprint key corresponding to the data to be deduplicated. Based on the identification information, a target storage area corresponding to the data to be deduplicated is determined. The target storage area corresponds to a preset instance. The preset instance is used to perform data write operations and / or data read operations on the target storage area. Then, based on the preset instance, a data read operation is performed to obtain the target storage data in the target storage area. Based on the target storage data, it is determined whether the data to be deduplicated exists. Thus, the target storage area corresponding to the data to be deduplicated can be determined through the identification information of the data to be deduplicated, and then the existence of the data to be deduplicated can be determined based on the target storage data in the target storage area, thereby improving the efficiency and accuracy of data deduplication.

[0053] In some embodiments of this disclosure, determining the target storage area corresponding to the deduplicated data based on the identification information in S120 may specifically include: performing hash processing on the first fingerprint key based on a preset hash algorithm to obtain the hash value corresponding to the deduplicated data; performing a remainder operation on the hash value and the preset total number of storage areas to obtain the address information of the storage area corresponding to the deduplicated data; and determining the preset storage area corresponding to the address information as the target storage area.

[0054] Optionally, the preset hash algorithm can be a hash code algorithm or any algorithm used to hash the first fingerprint key, without limitation.

[0055] In this embodiment of the disclosure, the specific implementation method of hashing the first fingerprint key based on a preset hash algorithm to obtain the hash value corresponding to the deduplicated data is similar to the existing implementation method of hashing data to obtain the hash value through a hash algorithm, and will not be described in detail here.

[0056] In this embodiment of the disclosure, the hash value and the preset total number of storage regions are moduloed, and the specific expression is as follows: hash(key)%region, where hash(key) represents the hash value obtained after hashing the first fingerprint key; and region represents the total number of storage regions.

[0057] The total number of storage areas is preset and stored in the electronic device, which can directly obtain the total number of storage areas.

[0058] Specifically, after obtaining the identification information of the data to be deduplicated, the electronic device performs hash processing on the first fingerprint key in the identification information using a preset hash algorithm to obtain the hash value corresponding to the data to be deduplicated. Then, it performs a remainder operation based on the hash value and the total number of storage areas, and determines the address information of the storage area corresponding to the data to be deduplicated based on the result of the remainder operation. Finally, it determines the target storage area from multiple storage areas based on the address information.

[0059] In this embodiment of the disclosure, the target storage area corresponding to the data to be deduplicated can be obtained based on the first fingerprint key in the identification information of the data to be deduplicated, ensuring the uniqueness and accuracy of the obtained target storage area.

[0060] In other embodiments of this disclosure, the identification information also includes time information corresponding to the data to be deduplicated. In step S120, determining the target storage area corresponding to the data to be deduplicated based on the identification information may specifically include: parsing the time information, using the parsing result as the address information of the storage area corresponding to the data to be deduplicated, and determining the preset storage area corresponding to the address information as the target storage area.

[0061] In this embodiment of the disclosure, the data to be deduplicated includes time information, such as the publication time and modification time of the data to be deduplicated.

[0062] Specifically, after obtaining the identification information corresponding to the data to be deduplicated, the electronic device parses the time information, including the year, month, and day. The parsing result obtained after parsing, with the time granularity as the day, is determined as the address information of the storage area corresponding to the data to be deduplicated. The address information of this storage area is also named with the time information. The preset storage area corresponding to the address information is determined as the target storage area. For example, if the parsing result of a certain data to be deduplicated is 20XX year X month XX day, then the storage area that matches the year, month, and day in the parsing result is determined as the target storage area.

[0063] In this embodiment of the disclosure, the target storage area corresponding to the data to be deduplicated can be obtained based on the time information corresponding to the data to be deduplicated, ensuring the uniqueness of the target storage area corresponding to each data to be deduplicated and improving the efficiency of obtaining the target storage area.

[0064] Based on the above embodiments of this disclosure, determining whether the deduplicated data exists based on the target stored data in S130 may specifically include: obtaining the second fingerprint key corresponding to the target stored data; comparing the first fingerprint key with the second fingerprint key to determine whether the deduplicated data exists.

[0065] In this embodiment of the disclosure, the second fingerprint key is the identification information carried when the target storage data is stored in the target storage area, and each target storage data corresponds to a second fingerprint key. The electronic device can directly obtain the second fingerprint key corresponding to the target storage data from the target storage area.

[0066] Specifically, the electronic device obtains the second fingerprint key corresponding to the target stored data in the target storage area and compares the first fingerprint key and the second fingerprint key. If any fingerprint key in the first fingerprint key and the second fingerprint key are the same, it is determined that the deduplicated data to be deduplicated already exists. If all fingerprint keys in the first fingerprint key and the second fingerprint key are different, it is determined that the deduplicated data to be deduplicated does not exist.

[0067] In this embodiment of the disclosure, the existence of the data to be deduplicated can be determined by comparing the second fingerprint key corresponding to the target stored data and the first fingerprint key corresponding to the data to be deduplicated, thereby improving the deduplication accuracy of the data to be deduplicated.

[0068] Furthermore, after determining whether the deduplicated data exists based on the target stored data, the data deduplication method may further include: if it is determined that the deduplicated data exists, feeding back the duplication result of the deduplicated data to the user; if it is determined that the deduplicated data does not exist, writing the deduplicated data to the target storage area and feeding back the writing result of the deduplicated data to the user.

[0069] In some embodiments of this disclosure, the preset instance includes a disk storage directory corresponding to the target storage area. Writing the deduplicated data to the target storage area may specifically include: performing a data writing operation based on the preset instance, writing the deduplicated data to the disk storage directory corresponding to the target storage area in the form of key-value pairs.

[0070] The key-value pair can be in the form of KV, where K represents the first fingerprint key corresponding to the deduplicated data and V represents the status flag bit of the deduplicated data.

[0071] In this embodiment of the disclosure, the status flag bit of the data to be deduplicated can be 1.

[0072] In this embodiment of the disclosure, when it is determined that the data to be deduplicated does not exist, a data writing operation is performed through a preset instance to write the data to be deduplicated to the disk storage directory corresponding to the target storage area in the form of key-value pairs. This ensures that the data is not duplicated, facilitates subsequent use, and reduces the data storage cost by storing it in the disk storage directory. It can also support deduplication of massive amounts of data, such as TB-level data.

[0073] In this embodiment of the disclosure, the target storage area includes multiple target files with preset data structures, each target file contains multiple index blocks, and each index block has an index range corresponding to the index block.

[0074] The default data structure can be an ordered string table (sstable) data structure.

[0075] The index range corresponding to an index block can be a pre-set range based on the storage space of the index block, etc., which allows for quick location and search of target stored data.

[0076] For example, the index range of index block 1 is key1 to key100, and the index range of index block 2 is key101 to key200. If the identification information of the target stored data is key5, then the target stored data can be located in index block 1 directly according to the identification information of the target stored data and the index range of the index block, so as to realize the fast location and search of the target stored data.

[0077] In this embodiment of the disclosure, when deduplicating data, the specified data can be quickly located by traversing the target files in the target storage area and based on the index blocks in the target files and the index values ​​corresponding to the index blocks. This accelerates the data query and write speed. At the same time, the ordered nature of the SSTable files ensures the arrangement order of the data in the files, making it easier to find the duplicate data fingerprint key when determining whether the data to be deduplicated exists based on the target storage data. Therefore, data deduplication can be performed effectively and efficiently on both a single large file and a large number of small files, improving the data deduplication speed.

[0078] Figure 2 This is a flowchart of another data deduplication method provided in this disclosure embodiment, such as... Figure 2 As shown, this data deduplication method may include the following steps:

[0079] S210. Obtain the weight data to be de-weighted and the corresponding identification information, including the first fingerprint key corresponding to the weight data to be de-weighted.

[0080] S220. Determine the target storage area corresponding to the deduplication data based on the identification information. The target storage area has a preset instance.

[0081] S230. Perform a data reading operation based on a preset instance to obtain the target storage data in the target storage area.

[0082] S240: Obtain the second fingerprint key corresponding to the target stored data, compare the first fingerprint key and the second fingerprint key to determine whether the deduplication data exists.

[0083] In this embodiment of the disclosure, step S250 is executed when it is determined that the data to be de-weighted exists, and step S260 is executed when it is determined that the data to be de-weighted does not exist.

[0084] S250: If it is confirmed that the data to be deduplicated exists, the duplicate results of the data to be deduplicated will be fed back to the user.

[0085] S260. If it is determined that the data to be deduplicated does not exist, the data to be deduplicated is written to the target storage area, and the writing result of the data to be deduplicated is fed back to the user.

[0086] It should be noted that the specific implementation methods of steps S210 to S260 in this embodiment are similar to those in the above embodiments, and will not be repeated here.

[0087] In this embodiment, when data deduplication is required or a data deduplication instruction is received, the system can acquire the data to be deduplicated and its corresponding identifier information. Based on the identifier information, it determines the target storage area corresponding to the data to be deduplicated (where the target storage area is a LevelDB database storage area). A data reading operation is performed based on a preset instance corresponding to the target storage area to obtain the target stored data. A second fingerprint key corresponding to the target stored data is acquired, and the first and second fingerprint keys are compared to determine if the data to be deduplicated exists. If either the first or second fingerprint key matches, the data to be deduplicated exists; otherwise, it does not exist. If the data to be deduplicated exists, the duplicate result is fed back to the user. If the data to be deduplicated does not exist, it is written to the target storage area based on the preset instance, and the writing result is fed back to the user. This fully utilizes the storage capacity and cost of the disk, achieving a more economical and efficient storage solution while ensuring persistent data storage and reliability, thus improving the efficiency and accuracy of data deduplication.

[0088] Figure 3 This is a schematic diagram of the structure of a data deduplication device provided in an embodiment of this disclosure.

[0089] In this embodiment, the data deduplication device can be installed within an electronic device and is understood as a functional module within the aforementioned electronic device. Specifically, the electronic device can be a server or a terminal, wherein the terminal specifically includes mobile phones, computers, or tablet computers, etc., without limitation.

[0090] like Figure 3 As shown, the data deweighting device 300 may include a data acquisition module 310, a region determination module 320, and a deweighting judgment module 330.

[0091] The data acquisition module 310 can be used to acquire the data to be de-weighted and the corresponding identification information, including the first fingerprint key corresponding to the data to be de-weighted.

[0092] The region determination module 320 can be used to determine the target storage region corresponding to the deduplicated data based on the identification information. The target storage region has a corresponding preset instance, which is used to perform data writing and / or data reading operations on the target storage region.

[0093] The deduplication judgment module 330 can be used to perform data reading operations based on a preset instance to obtain the target storage data in the target storage area, and determine whether the data to be deduplicated exists based on the target storage data.

[0094] In this embodiment, it is possible to obtain the data to be deduplicated and the corresponding identification information. The identification information includes a first fingerprint key corresponding to the data to be deduplicated. Based on the identification information, a target storage area corresponding to the data to be deduplicated is determined. The target storage area corresponds to a preset instance. The preset instance is used to perform data write operations and / or data read operations on the target storage area. Then, based on the preset instance, a data read operation is performed to obtain the target storage data in the target storage area. Based on the target storage data, it is determined whether the data to be deduplicated exists. Thus, the target storage area corresponding to the data to be deduplicated can be determined through the identification information of the data to be deduplicated, and then the existence of the data to be deduplicated can be determined based on the target storage data in the target storage area, thereby improving the efficiency and accuracy of data deduplication.

[0095] In some embodiments of this disclosure, the region determination module 320 may be specifically used to perform hash processing on the first fingerprint key based on a preset hash algorithm to obtain the hash value corresponding to the deduplicated data; perform a remainder operation on the hash value and the preset total number of storage regions to obtain the address information of the storage region corresponding to the deduplicated data; and determine the preset storage region corresponding to the address information as the target storage region.

[0096] In some embodiments of this disclosure, the identification information also includes time information corresponding to the data to be deduplicated.

[0097] The region determination module 320 can also be specifically used to parse and process time information, use the parsing and processing results as the address information of the storage area corresponding to the data to be deduplicated, and determine the preset storage area corresponding to the address information as the target storage area.

[0098] In some embodiments of this disclosure, the deduplication judgment module 330 may be specifically used to obtain the second fingerprint key corresponding to the target stored data; compare the first fingerprint key with the second fingerprint key to determine whether the data to be deduplicated exists.

[0099] In some embodiments of this disclosure, the data de-duplication device 300 may include a first feedback module and a second feedback module.

[0100] The first feedback module can be used to provide the user with the duplicate results of the data to be deduplicated when it is determined that the data to be deduplicated exists.

[0101] The second feedback module can be used to write the deduplication data to the target storage area when it is determined that the data to be deduplicated does not exist, and to feed back the writing result of the deduplication data to the user.

[0102] In some embodiments of this disclosure, the preset instance includes the disk storage directory corresponding to the target storage area.

[0103] The second feedback module can be specifically used to perform data writing operations based on a preset instance, writing the deduplicated data to be written to the disk storage directory corresponding to the target storage area in the form of key-value pairs.

[0104] In some embodiments of this disclosure, the target storage area includes multiple target files with preset data structures, each target file contains multiple index blocks, and each index block has an index range corresponding to the index block.

[0105] It should be noted that, Figure 3 The data deduplication device 300 shown can execute the various steps in the above method embodiments and achieve the various processes and effects in the above method embodiments, which will not be elaborated here.

[0106] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0107] In this embodiment of the disclosure, Figure 4 The electronic devices shown can be servers or terminals, where terminals specifically include mobile phones, computers, or tablets, etc., without limitation.

[0108] like Figure 4 As shown, the electronic device may include a processor 410 and a memory 420 storing computer program instructions.

[0109] Specifically, the processor 410 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this disclosure.

[0110] Memory 420 may include mass storage for information or instructions. For example, and not limitingly, memory 420 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 420 may include removable or non-removable (or fixed) media. Where appropriate, memory 420 may be internal or external to the integrated gateway device. In a particular embodiment, memory 420 is non-volatile solid-state memory. In a particular embodiment, memory 420 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0111] The processor 410 reads and executes computer program instructions stored in the memory 420 to perform the steps of the data deduplication method provided in the embodiments of this disclosure.

[0112] In one example, the electronic device may also include a transceiver 430 and a bus 440. Wherein, as... Figure 4 As shown, the processor 410, memory 420 and transceiver 430 are connected via bus 440 and communicate with each other.

[0113] Bus 440 may include hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 440 may include one or more buses.

[0114] This disclosure also provides a computer-readable storage medium that can store a computer program that, when executed by a processor, enables the processor to implement the data deduplication method provided in this disclosure.

[0115] The aforementioned storage medium may include, for example, a memory 420 containing computer program instructions, which can be executed by a processor 410 of an electronic device to perform the data deduplication method provided in this embodiment. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0116] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data deduplication method, characterized in that, include: Obtain the weight data to be sorted and the identification information corresponding to the weight data to be sorted, wherein the identification information includes the first fingerprint key corresponding to the weight data to be sorted; Based on the identification information, the target storage area corresponding to the deduplication data is determined. The target storage area has a preset instance, which is used to perform data writing and / or data reading operations on the target storage area. Based on the preset instance, a data reading operation is performed to obtain the target storage data in the target storage area, and based on the target storage data, it is determined whether the data to be deduplicated exists. The step of determining the target storage area corresponding to the deduplication data based on the identification information includes: The first fingerprint key is hashed based on a preset hash algorithm to obtain the hash value corresponding to the deduplicated data; The hash value and the preset total number of storage areas are moduloed to obtain the address information of the storage area corresponding to the data to be deduplicated; The preset storage area corresponding to the address information is determined as the target storage area.

2. The method according to claim 1, characterized in that, The identification information also includes the time information corresponding to the weight data to be sorted; The step of determining the target storage area corresponding to the deduplication data based on the identification information includes: The time information is parsed and processed, and the parsing result is used as the address information of the storage area corresponding to the data to be deduplicated. The preset storage area corresponding to the address information is determined as the target storage area.

3. The method according to claim 1, characterized in that, Determining whether the deduplication data exists based on the target stored data includes: Obtain the second fingerprint key corresponding to the target stored data; The first fingerprint key is compared with the second fingerprint key to determine whether the deduplication data exists.

4. The method according to claim 1, characterized in that, After determining whether the deduplication data exists based on the target stored data, the method further includes: If the existence of the data to be deduplicated is confirmed, the duplication result of the data to be deduplicated will be fed back to the user. If it is determined that the data to be deduplicated does not exist, the data to be deduplicated is written to the target storage area, and the writing result of the data to be deduplicated is fed back to the user.

5. The method according to claim 4, characterized in that, The preset instance includes the disk storage directory corresponding to the target storage area, and the step of writing the deduplication data to the target storage area includes: Based on the preset instance, a data writing operation is performed to write the deduplicated data to the disk storage directory corresponding to the target storage area in the form of key-value pairs.

6. The method according to claim 1, characterized in that, The target storage area includes multiple target files with preset data structures. Each target file contains multiple index blocks, and each index block has an index range corresponding to the index block.

7. A data deduplication device, characterized in that, include: The data acquisition module is used to acquire the weight data to be sorted and the identification information corresponding to the weight data to be sorted, wherein the identification information includes a first fingerprint key corresponding to the weight data to be sorted. The region determination module is used to determine the target storage region corresponding to the deduplication data based on the identification information. The target storage region has a preset instance, and the preset instance is used to perform data writing operations and / or data reading operations on the target storage region. The deduplication determination module is used to perform a data reading operation based on the preset instance to obtain the target storage data in the target storage area, and to determine whether the data to be deduplicated exists based on the target storage data. The step of determining the target storage area corresponding to the deduplication data based on the identification information includes: The first fingerprint key is hashed based on a preset hash algorithm to obtain the hash value corresponding to the deduplicated data; The hash value and the preset total number of storage areas are moduloed to obtain the address information of the storage area corresponding to the data to be deduplicated; The preset storage area corresponding to the address information is determined as the target storage area.

8. An electronic device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the data deduplication method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the data deduplication method according to any one of claims 1-6.

Images