Data deduplication method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a data deduplication method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: under the condition that data to be subjected to duplicate removal is obtained, inserting the data to be subjected to duplicate removal into a first hash table for performing hash duplicate removal until the data in the first hash table reaches a target number of rows, and outputting the data in the first target number of rows in the first hash table; under the condition that the first hash table is not full, continuing to insert the data to be subjected to duplicate removal into the first hash table for performing hash duplicate removal until the first hash table is full; under the condition that the first hash table is full, repeated detection is carried out on the data to be subjected to duplicate removal and existing data in the first hash table, and the data to be subjected to duplicate removal, which are judged to be non-repeated, are subjected to disk brushing; and under the condition that the processing of the data to be subjected to duplicate removal is completed, performing duplicate removal processing on all the data in the disk, outputting the data, and outputting the data except the data in the previous target number of rows in the first hash table. According to the scheme, the previous target number of rows of data obtained through duplicate removal can be returned in advance, and the user experience can be improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data deduplication method, device, electronic device, storage medium and product. Background Art

[0002] In relational databases, deduplication is a commonly used data processing operation, which is mainly used to eliminate redundant records and ensure data consistency and accuracy.

[0003] In the prior art, data deduplication can be performed by constructing and using a hash table that only stores non-duplicate data, and then the deduplication result is output after all data is deduplicated. However, in this way, users need to wait for a long time to get the deduplication result, which has a poor user experience. Summary of the invention

[0004] The present invention provides a data deduplication method, device, electronic device, storage medium and product, which can return the previous target number of rows of data obtained by deduplication in advance, helping to improve user experience.

[0005] In a first aspect, an embodiment of the present invention provides a data deduplication method, comprising:

[0006] When the data to be deduplicated is obtained, the data to be deduplicated is inserted into the first hash table for hash deduplication, until the data in the first hash table reaches a target number of rows, and then the first target number of rows of data in the first hash table is output;

[0007] When the first hash table is not full, the data to be deduplicated is continuously inserted into the first hash table for hash deduplication until the first hash table is full;

[0008] When the first hash table is full, the to-be-deduplicated data and the existing data in the first hash table are subjected to duplication detection, and the to-be-deduplicated data that is determined to be non-duplicate is flushed to disk;

[0009] When the processing of the data to be deduplicated is completed, all the data on the disk is deduplicated and then output, and the data in the first hash table except for the first target number of rows of data is output.

[0010] Furthermore, the first hash table can accommodate a target number of rows of data at most. Accordingly, after outputting the first target number of rows of data in the first hash table, the method further includes:

[0011] Perform duplication detection on the data to be deduplicated and the existing data in the first hash table, insert the data to be deduplicated that are determined to be non-duplicate into the second hash table for hash deduplication, and flush the data in the second hash table to disk after the second hash table is full;

[0012] Continuously performing hash deduplication in the second hash table and flushing the data in the second hash table to disk until the processing of the data to be deduplicated is completed;

[0013] All data on disk is deduplicated and then output.

[0014] Furthermore, all data on the disk is deduplicated, including:

[0015] Multiple ordered disk flushing data fragments corresponding to all the data on the disk are merged and deduplicated.

[0016] Furthermore, the destruction scenario of the first hash table includes one of the following:

[0017] The second hash table has never triggered a disk flush and the processing of the to-be-deduplicated data has been completed;

[0018] The second hash table has triggered disk flushing at least once, and before deduplication processing is performed on all data on the disk after the deduplication data processing is completed.

[0019] Furthermore, the first hash table can accommodate more than a target number of rows of data at most. Accordingly, after outputting the first target number of rows of data in the first hash table, the method further includes:

[0020] Continue inserting the to-be-deduplicated data into the first hash table for hash deduplication until the first hash table is full;

[0021] When the first hash table is full, the data to be deduplicated is detected for duplication with the existing data in the first hash table, and the data to be deduplicated that is determined to be non-duplicate is inserted into the second hash table for hash deduplication, until the second hash table is full and the data in the second hash table is flushed to disk;

[0022] Continuously performing hash deduplication in the second hash table and flushing the data in the second hash table to disk until the processing of the data to be deduplicated is completed;

[0023] All data on the disk are deduplicated and then output, and data in the first hash table except for the first target number of rows of data are output.

[0024] Furthermore, the to-be-deduplicated data is acquired in batches from the lower-level operator nodes of the deduplication operator, and the capacity of the first hash table is at least greater than the amount of data acquired in one batch of the to-be-deduplicated data.

[0025] In a second aspect, an embodiment of the present invention provides a data deduplication device, comprising:

[0026] A first processing module is used to, when obtaining the data to be deduplicated, insert the data to be deduplicated into a first hash table for hash deduplication, until the data in the first hash table reaches a target number of rows, and then output the first target number of rows of data in the first hash table;

[0027] A second processing module, configured to, when the first hash table is not full, continue to insert the to-be-deduplicated data into the first hash table for hash deduplication until the first hash table is full;

[0028] A third processing module is used for performing duplication detection on the to-be-deduplicated data and the existing data in the first hash table when the first hash table is full, and flushing the to-be-deduplicated data that is determined to be non-duplicate to the disk;

[0029] The fourth processing module is used to deduplicate all the data on the disk and then output them after the deduplication processing is completed, and output the data in the first hash table except the first target number of rows of data.

[0030] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0031] at least one processor; and

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to the first aspect.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method described in the first aspect when executed.

[0035] In a fifth aspect, an embodiment of the present invention provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method as described in the first aspect is implemented.

[0036] The technical solution of the embodiment of the present invention can return the first target number of rows of data obtained by deduplication in advance during the process of deduplication of data to be deduplicated. Returning part of the data in advance helps to improve user experience; and returning part of the data in advance can meet the usage requirements in specific scenarios, such as the scenario where the deduplication operation may not need to be fully executed and can be terminated in advance by returning part of the data, thereby optimizing the usage efficiency.

[0037] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 is a flow chart of a data deduplication method provided according to Embodiment 1 of the present invention;

[0040] Figure 2 is a structural schematic diagram of a data deduplication device provided according to Embodiment 2 of the present invention;

[0041] Figure 3 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] Embodiment 1

[0045] Figure 1This is a flow chart of a data deduplication method provided according to Embodiment 1 of the present invention. This embodiment is applicable to the case where part of the data is returned in advance during the data deduplication process. The method can be executed by a data deduplication device, which can be implemented in the form of software and / or hardware and integrated in an electronic device. Furthermore, the electronic device includes but is not limited to: a computer, a laptop computer, etc.

[0046] like Figure 1 As shown, the method includes:

[0047] S110. When the data to be deduplicated is obtained, the data to be deduplicated is inserted into a first hash table for hash deduplication. When the data in the first hash table reaches a target number of rows, the first target number of rows of data in the first hash table is output.

[0048] The data to be deduplicated may be data to be deduplicated, and deduplication is mainly used to eliminate redundant records to ensure data consistency and accuracy. There is no limitation on the source and acquisition method of the data to be deduplicated, as long as the data to be deduplicated can be obtained for deduplication.

[0049] The first hash table may be a hash table for performing hash deduplication on the data to be deduplicated. The structure of the first hash table is not limited, and may be a single hash table structure or a composite structure of multiple single hash tables. During the deduplication process, the first hash table may reside in the memory of the electronic device.

[0050] In one embodiment, the to-be-deduplicated data is acquired in batches from the lower-level operator nodes of the deduplication operator, and the capacity of the first hash table is at least greater than the amount of data acquired in one batch of the to-be-deduplicated data.

[0051] The deduplication operator may be an operator for performing deduplication tasks. The lower operator node of the deduplication operator may be a data processing or data source node at a lower level relative to the deduplication operator, and may be any type of data provider, such as a database query, etc. The lower operator node is responsible for obtaining the data to be deduplicated and passing it to the deduplication operator for further processing.

[0052] In an embodiment of the present invention, the data to be deduplicated can be obtained in batches through the lower operator nodes of the deduplication operator. The batch acquisition can effectively control memory usage and facilitate resource management. The capacity of the first hash table is at least greater than the amount of data to be deduplicated obtained in one batch, that is, the capacity of the first hash table can at least guarantee the processing of one batch of data to be deduplicated. Among them, the amount of data to be deduplicated obtained in one batch is not limited and can be set according to actual application needs.

[0053] In an embodiment of the present invention, the first hash table can be empty when initialized. When the data to be deduplicated is obtained in each batch, the data to be deduplicated is inserted into the first hash table for hash deduplication, which can be specifically understood as follows: for each data unit of the data to be deduplicated, its hash value is calculated by a hash function; for each hash value, check whether there is a corresponding slot in the first hash table; if the corresponding slot is empty or the current hash value is not registered in the table, it is considered to be a new data unit, and its hash value is added to the first hash table, while retaining its original data for subsequent processing; if there is a corresponding value in the hash table, indicating that this is a duplicate data unit, it will be discarded without being inserted into the first hash table.

[0054] By continuously inserting data into the first hash table in the above manner, until the data in the first hash table reaches the target number of rows, the first target number of rows of data in the first hash table are output, that is, the original data corresponding to the first target number of rows of data in the first hash table are output as the deduplicated data. Instead of outputting data after all the deduplication of the data to be deduplicated is completed, the first target number of rows of data in the first hash table are returned in advance during the deduplication process. Among them, the target number of rows can be set according to the actual application needs and is not limited here.

[0055] It should be noted that, when the first target number of rows of data in the first hash table are returned in advance, this part of the data returned in advance will not be discarded, but will continue to participate in the subsequent hash deduplication, but this part of the data will not be output later.

[0056] S120: When the first hash table is not full, continue to insert the data to be deduplicated into the first hash table for hash deduplication until the first hash table is full.

[0057] In actual applications, there is a situation where the data in the first hash table reaches the target number of rows, but the first hash table is not full. In this case, the acquired data to be deduplicated is continuously inserted into the first hash table for hash deduplication until the first hash table is full. The method of performing hash deduplication in the first hash table in this step is basically the same as that in S110 above, and will not be repeated here.

[0058] S130. When the first hash table is full, perform duplication detection on the data to be deduplicated and the existing data in the first hash table, and flush the data to be deduplicated that is determined to be non-duplicate to the disk.

[0059] In this step, when the first hash table is full, the data to be deduplicated and the existing data in the first hash table are detected for duplication. Specifically: it is determined whether the hash value corresponding to the data to be deduplicated has a corresponding slot in the first hash table. If so, it is determined to be duplicate data and discarded; if not, it is determined to be non-duplicate data to be deduplicated, and the data to be deduplicated determined to be non-duplicate is flushed to disk. The method of flushing is not limited here.

[0060] It should be noted that the first hash table can be a hash table that can accommodate a maximum of a target number of rows of data, and the capacity of the first hash table can be determined by the target number. The larger the target number, the larger the capacity. In this case, after S110 is executed, the first hash table is full, and all the data in the first hash table, that is, the data of the previous target number of rows, is output in advance. And after S110 is executed, there is no need to execute S120, but directly execute S130. The advantage of this setting is that some data can be returned in advance in a memory-constrained scenario.

[0061] It should be noted that the first hash table can be a hash table that can accommodate more than the target number of rows of data. In this case, after S110 is executed, the first hash table is not full, and the first target number of rows of data in the first hash table are output in advance. After S110 is executed, S120 and S30 are executed sequentially. The advantage of this setting is that data output is performed when the target number of rows of data is reached, and some data can be returned in advance as soon as possible.

[0062] S140. When the deduplication processing of the data to be deduplicated is completed, all the data on the disk are deduplicated and then output, and the data in the first hash table except for the first target number of rows of data is output.

[0063] In this step, after the deduplication data processing is completed, that is, all the deduplication data have participated in the repeatability detection in the first hash table, all the data that have been flushed in the disk are deduplicated and then output. This deduplication method is not limited. In addition, if there is other data in the first hash table besides the previous target number of rows of data, then this part of data is output; if there is no other data in the first hash table besides the previous target number of rows of data, that is, all the data in the first hash table has been output in advance, then there is no need to output the data in the first hash table in this step.

[0064] There is no limitation on the method of deduplication processing for all the data that has been flushed to the disk, because the method of deduplication processing may be related to the method of flushing. For example, if data sharding is performed during the flushing process, the subsequent deduplication of the flushed data also needs to process the sharded data accordingly.

[0065] The technical solution of the embodiment of the present invention can return the first target number of rows of data obtained by deduplication in advance during the process of deduplication of data to be deduplicated. Returning part of the data in advance helps to improve user experience; and returning part of the data in advance can meet the usage requirements in specific scenarios, such as the scenario where the deduplication operation may not need to be fully executed and can be terminated in advance by returning part of the data, thereby optimizing the usage efficiency.

[0066] The above-mentioned deduplication process of the data to be deduplicated is implemented through the first hash table. The following combines the first hash table and the second hash table, that is, the mechanism of multi-hash table deduplication, to deduplicate the data to be deduplicated. Among them, the second hash table can be a hash table used for hash deduplication of the data to be deduplicated, and the second hash table and the first hash table are different hash tables. The structure of the second hash table is not limited, and it can be a single hash table structure or a composite structure of multiple single hash tables. During the deduplication process, the second hash table will not reside in the memory of the electronic device, but the data therein can be flushed and new data can be inserted.

[0067] In one embodiment, the first hash table can accommodate a target number of rows of data at most. Accordingly, after outputting the first target number of rows of data in the first hash table, the method further includes:

[0068] Perform duplication detection on the data to be deduplicated and the existing data in the first hash table, insert the data to be deduplicated that are determined to be non-duplicate into the second hash table for hash deduplication, and flush the data in the second hash table to disk after the second hash table is full;

[0069] Continuously performing hash deduplication in the second hash table and flushing the data in the second hash table to disk until the processing of the data to be deduplicated is completed;

[0070] All data on disk is deduplicated and then output.

[0071] When the first hash table can accommodate a maximum of a target number of rows of data, when the first hash table is not full, the data to be deduplicated is inserted into the first hash table for hash deduplication. When the data in the first hash table reaches the target number of rows, that is, the first hash table is full, the first target number of rows of data in the first hash table is output, that is, all the data in the first hash table is output.

[0072] After that, the obtained data to be deduplicated is first detected for duplication with the existing data in the first hash table, the data determined to be duplicated is discarded, and the data to be deduplicated that is determined to be non-duplicate is inserted into the second hash table initialized to be empty. When inserting into the second hash table, the data must also be hashed and deduplicated in the second hash table before being inserted. When the second hash table is full, the data in the second hash table needs to be flushed to disk first, and the method of flushing is not limited, and then the second hash table is cleared.

[0073] The data to be deduplicated that is subsequently obtained will continue to be detected for duplication with the existing data in the first hash table and inserted into the second hash table for hash deduplication. After the second hash table is full, the disk will be refreshed again until the processing of the data to be deduplicated is completed. That is, all the data to be deduplicated have participated in duplication detection in the first hash table and the second hash table. After that, all the data on the disk are deduplicated and output, and the deduplication method is not limited.

[0074] In one embodiment, deduplication processing is performed on all data in the disk, including: merging and deduplication of multiple ordered disk flushing data fragments corresponding to all data in the disk.

[0075] Each time the data in the second hash table is flushed to disk, the data can be flushed in an orderly manner according to the arrangement order of the data in the second hash table, so that multiple ordered flushing data fragments can be formed on the disk. The multiple ordered flushing data fragments in the disk are merged and deduplicated because although a single ordered flushing data fragment comes from data that has been deduplicated by hashing based on the second hash table, there may still be duplicate values ​​between different flushing data fragments. There is no limitation on the method of merging and deduplicating.

[0076] In one embodiment, the destruction scenarios of the first hash table include one of the following: the second hash table has never triggered a disk flush and the processing of the data to be deduplicated is completed; the second hash table has triggered a disk flush at least once, and before all data on the disk is deduplicated after the processing of the data to be deduplicated is completed.

[0077] Among them, the second hash table has never triggered a disk flush and the processing of the data to be deduplicated has been completed. It can be understood that all the data to be deduplicated have participated in the repeatability detection in the first hash table and the second hash table. The capacity of the second hash table is sufficient to complete the processing of all the data to be deduplicated without flushing the disk. At this time, the first hash table can be destroyed.

[0078] Among them, the second hash table has triggered a disk flush at least once, and before all the data on the disk are deduplicated after the deduplication processing is completed, it can be understood that all the data to be deduplicated have participated in the duplication detection in the first hash table and the second hash table. The second hash table can then destroy the first hash table after flushing the data therein, and then deduplicate all the data on the disk.

[0079] In one embodiment, the first hash table can accommodate more than a target number of rows of data at most. Accordingly, after outputting the first target number of rows of data in the first hash table, the method further includes:

[0080] Continue inserting the to-be-deduplicated data into the first hash table for hash deduplication until the first hash table is full;

[0081] When the first hash table is full, the data to be deduplicated is detected for duplication with the existing data in the first hash table, and the data to be deduplicated that is determined to be non-duplicate is inserted into the second hash table for hash deduplication, until the second hash table is full and the data in the second hash table is flushed to disk;

[0082] Continuously performing hash deduplication in the second hash table and flushing the data in the second hash table to disk until the processing of the data to be deduplicated is completed;

[0083] All data on the disk are deduplicated and then output, and data in the first hash table except for the first target number of rows of data are output.

[0084] In the case that the first hash table can accommodate more than the target number of rows of data, when the data in the first hash table does not reach the target number of rows of data, the data to be deduplicated is inserted into the first hash table for hash deduplication until the data in the first hash table reaches the target number of rows, and the first target number of rows of data in the first hash table are output. At this time, the first hash table is not full, and the obtained data to be deduplicated continues to be inserted into the first hash table for hash deduplication until the first hash table is full.

[0085] When the first hash table is full, the acquired data to be deduplicated is first detected for duplication with the existing data in the first hash table, the data determined to be duplicated is discarded, and the data to be deduplicated that is determined to be non-duplicate is inserted into the second hash table initialized to be empty. When inserting into the second hash table, the data must also be hashed and deduplicated in the second hash table before being inserted. When the second hash table is full, the data in the second hash table needs to be flushed to disk first, and the method of flushing is not limited, and then the second hash table is cleared.

[0086] The data to be deduplicated obtained later will continue to be detected for duplication with the existing data in the first hash table and inserted into the second hash table for hash deduplication. When the second hash table is full, the disk will be refreshed again until the deduplication data processing is completed, that is, all the deduplication data have participated in duplication detection in the first hash table and the second hash table. All the data on the disk will be deduplicated and then output, and the data except the first target number of rows in the first hash table will be output.

[0087] Embodiment 2

[0088] Figure 2 is a schematic diagram of the structure of a data deduplication device provided according to the second embodiment of the present invention. This embodiment is applicable to the case where part of the data is returned in advance during the data deduplication process, such as Figure 2 As shown, the specific structure of the device includes:

[0089] The first processing module 21 is used for inserting the to-be-deduplicated data into the first hash table for hash deduplication when the to-be-deduplicated data is obtained, and outputting the first target number of rows of data in the first hash table when the data in the first hash table reaches the target number of rows;

[0090] A second processing module 22 is used for continuing to insert the to-be-deduplicated data into the first hash table for hash deduplication when the first hash table is not full, until the first hash table is full;

[0091] The third processing module 23 is used for performing duplication detection on the to-be-deduplicated data and the existing data in the first hash table when the first hash table is full, and flushing the to-be-deduplicated data that is determined to be non-duplicate to the disk;

[0092] The fourth processing module 24 is used to deduplicate all the data on the disk and then output them after the deduplication processing is completed, and output the data in the first hash table except the first target number of rows of data.

[0093] The data deduplication device provided in this embodiment, when the first processing module obtains the data to be deduplicated, inserts the data to be deduplicated into the first hash table for hash deduplication, until the data in the first hash table reaches the target number of rows, and outputs the first target number of rows of data in the first hash table; when the first hash table is not full, the second processing module continues to insert the data to be deduplicated into the first hash table for hash deduplication until the first hash table is full; when the first hash table is full, the third processing module performs repetitive detection on the data to be deduplicated and the existing data in the first hash table, and flushes the data to be deduplicated that is determined to be non-duplicate; when the processing of the data to be deduplicated is completed, the fourth processing module performs deduplication processing on all the data on the disk and outputs it, and outputs the data except the first target number of rows of data in the first hash table. In the process of deduplication of the data to be deduplicated, the scheme can return the first target number of rows of data obtained by deduplication in advance, and returning part of the data in advance helps to improve the user experience; and returning part of the data in advance can meet the use requirements in specific scenarios, such as the scenario where the deduplication operation may not need to be fully executed, and only part of the data needs to be returned to end in advance, thereby optimizing the use efficiency.

[0094] Furthermore, the first hash table can accommodate a maximum of a target number of rows of data, and accordingly, the device further includes:

[0095] A fifth processing module is used to perform duplication detection on the data to be deduplicated and the existing data in the first hash table after outputting the first target number of rows of data in the first hash table, insert the data to be deduplicated that are determined to be non-duplicate into the second hash table for hash deduplication, and flush the data in the second hash table to disk after the second hash table is full;

[0096] a sixth processing module, configured to continuously perform hash deduplication in the second hash table and flush the data in the second hash table to a disk until the processing of the data to be deduplicated is completed;

[0097] The seventh processing module is used to deduplicate all the data in the disk and then output it.

[0098] Furthermore, the seventh processing module is specifically used for:

[0099] Multiple ordered disk flushing data fragments corresponding to all the data on the disk are merged and deduplicated.

[0100] Furthermore, the destruction scenario of the first hash table includes one of the following:

[0101] The second hash table has never triggered a disk flush and the processing of the to-be-deduplicated data has been completed;

[0102] The second hash table has triggered disk flushing at least once, and before deduplication processing is performed on all data on the disk after the deduplication data processing is completed.

[0103] Furthermore, the first hash table can accommodate a maximum of more than a target number of rows of data, and accordingly, the device further includes:

[0104] An eighth processing module, configured to, after outputting the first target number of rows of data in the first hash table, continue to insert the to-be-deduplicated data into the first hash table for hash deduplication until the first hash table is full;

[0105] a ninth processing module, configured to, when the first hash table is full, perform duplication detection on the data to be deduplicated and the existing data in the first hash table, insert the data to be deduplicated that is determined to be non-duplicate into the second hash table for hash deduplication, and flush the data in the second hash table to disk after the second hash table is full;

[0106] a tenth processing module, configured to continuously perform hash deduplication in the second hash table and flush the data in the second hash table to a disk until the processing of the data to be deduplicated is completed;

[0107] An eleventh processing module is used for performing deduplication processing on all data on the disk and then outputting the deduplication processing, and outputting the data in the first hash table except the first target number of rows of data.

[0108] Furthermore, the to-be-deduplicated data is acquired in batches from the lower-level operator nodes of the deduplication operator, and the capacity of the first hash table is at least greater than the amount of data acquired in one batch of the to-be-deduplicated data.

[0109] The data deduplication device provided in the embodiment of the present invention can execute the data deduplication method provided in the embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0110] Embodiment 3

[0111] Figure 3 Schematic diagram of the structure of the electronic device implementing the embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0112] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0113] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0114] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data deduplication method.

[0115] In some embodiments, the data deduplication method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data deduplication method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data deduplication method in any other appropriate manner (e.g., by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0118] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0121] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data deduplication method, characterized in that: include: When the data to be deduplicated is obtained, the data to be deduplicated is inserted into the first hash table for hash deduplication, until the data in the first hash table reaches a target number of rows, and then the first target number of rows of data in the first hash table is output; When the first hash table is not full, the data to be deduplicated is continuously inserted into the first hash table for hash deduplication until the first hash table is full; When the first hash table is full, the to-be-deduplicated data and the existing data in the first hash table are subjected to duplication detection, and the to-be-deduplicated data that is determined to be non-duplicate is flushed to disk; When the processing of the data to be deduplicated is completed, all the data on the disk is deduplicated and then output, and the data in the first hash table except for the first target number of rows of data is output.

2. The method according to claim 1, characterized in that The first hash table can accommodate a target number of rows of data at most. Accordingly, after outputting the first target number of rows of data in the first hash table, the method further includes: Perform duplication detection on the data to be deduplicated and the existing data in the first hash table, insert the data to be deduplicated that are determined to be non-duplicate into the second hash table for hash deduplication, and flush the data in the second hash table to disk after the second hash table is full; Continuously performing hash deduplication in the second hash table and flushing the data in the second hash table to disk until the processing of the data to be deduplicated is completed; All data on disk is deduplicated and then output.

3. The method according to claim 2, characterized in that Deduplication of all data on disk, including: Multiple ordered disk flushing data fragments corresponding to all the data on the disk are merged and deduplicated.

4. The method according to claim 2, characterized in that: The destruction scenario of the first hash table includes one of the following: The second hash table has never triggered a disk flush and the processing of the to-be-deduplicated data has been completed; The second hash table has triggered disk flushing at least once, and before deduplication processing is performed on all data on the disk after the deduplication data processing is completed.

5. The method according to claim 1, characterized in that The first hash table can accommodate more than a target number of rows of data at most. Accordingly, after outputting the first target number of rows of data in the first hash table, the method further includes: Continue inserting the to-be-deduplicated data into the first hash table for hash deduplication until the first hash table is full; When the first hash table is full, the data to be deduplicated is detected for duplication with the existing data in the first hash table, and the data to be deduplicated that is determined to be non-duplicate is inserted into the second hash table for hash deduplication, until the second hash table is full and the data in the second hash table is flushed to disk; Continuously performing hash deduplication in the second hash table and flushing the data in the second hash table to disk until the processing of the data to be deduplicated is completed; All data on the disk are deduplicated and then output, and data in the first hash table except for the first target number of rows of data are output.

6. The method according to claim 1, characterized in that The to-be-deduplicated data is acquired in batches from the lower-level operator nodes of the deduplication operator, and the capacity of the first hash table is at least greater than the amount of data of the to-be-deduplicated data acquired in one batch.

7. A data deduplication device, characterized in that: include: A first processing module is used to, when obtaining the data to be deduplicated, insert the data to be deduplicated into a first hash table for hash deduplication, until the data in the first hash table reaches a target number of rows, and then output the first target number of rows of data in the first hash table; A second processing module, configured to, when the first hash table is not full, continue to insert the to-be-deduplicated data into the first hash table for hash deduplication until the first hash table is full; A third processing module is used for performing duplication detection on the to-be-deduplicated data and the existing data in the first hash table when the first hash table is full, and flushing the to-be-deduplicated data that is determined to be non-duplicate to the disk; The fourth processing module is used to deduplicate all the data on the disk and then output them after the deduplication processing is completed, and output the data in the first hash table except the first target number of rows of data.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images