Column type repeated data deletion method, device and equipment

Abstract

The embodiment of the invention provides a column type repeated data deleting method, device and equipment. The method comprises the steps that input data is received, and a column type storage file in a column type storage unit is inquired to determine updated column data; performing data paging on the updated column data according to columns to obtain updated column paging data, and determining an updated column paging index of each updated column paging data; querying each updated column paging index from a column index unit; if the updated column paging index is queried, deleting the corresponding column paging data, taking the corresponding updated column paging index as a deleted column paging index, and adding the deleted column paging index into column metadata of the column storage file to obtain an updated column storage file; and if the updated column type storage file is not queried, reserving the column paging data corresponding to the updated column paging index, and obtaining the updated column type storage file. According to the method, the deletion of multiple copies of the same column, the deletion of the same data of different columns and the deletion of the same data under different compression or codes can be realized, the deduplication rate is improved, and the influence of additional storage and throughput rate is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a column-based duplicate data deleting method, device and equipment. Background Art

[0002] The rapid development of the Hadoop ecosystem has led to the emergence of open data lake storage formats such as Hudi. These high-performance columnar storage formats provide online analytical processing (OLAP) capabilities for massive amounts of data, meeting the growing demand for time-sensitive data analysis. Hadoop is a distributed system infrastructure, while Hudi is an open data lake table format that stores data in a columnar format and provides the ability to update, delete, and consume changed data on top of the Hadoop file system.

[0003] Research shows that up to 60% of the data stored in application systems is redundant. This percentage increases over time, and ensuring reliable and durable data storage can require more than 10 times the storage space and management costs. Consequently, the issue of high data redundancy in storage systems has attracted increasing attention, and reducing storage capacity has become a hot research topic. Data deduplication is one such capacity optimization technology. By eliminating duplicate data in storage systems, it reduces the amount of data actually stored or transmitted over the network. It has been widely used in backup, long-term archiving, and data disaster recovery.

[0004] Existing deduplication technologies include: full file detection, block detection based on fixed block algorithms, content-based variable-length block detection, and sliding block detection. However, these deduplication technologies suffer from drawbacks such as the inability to detect redundancy in certain scenarios, reduced deduplication rates in certain scenarios, and reduced input / output (I / O) efficiency. Summary of the Invention

[0005] The embodiments of the present invention provide a column-based data deduplication method, apparatus, and device, which improve the deduplication rate and reduce the impact of additional storage and throughput.

[0006] In a first aspect, this embodiment provides a column-based data deduplication method, the method comprising:

[0007] receiving input data, and querying a column storage file in a column storage unit to determine updated column data, wherein the column storage unit is used to store a column storage file including column data and column metadata;

[0008] Paging the update column data by column to obtain at least one update column paging data, and determining the update column paging index of each update column paging data;

[0009] querying each of the updated column paging indexes from a column index unit, wherein the column index unit is used to store column paging index metadata including a column paging index and a storage location of corresponding column paging data;

[0010] If the update column paging index is found, the column paging data corresponding to the update column paging index is deleted, and the update column paging index corresponding to the deleted column paging data is used as the deletion column paging index and added to the column metadata of the column storage file to obtain the updated column storage file;

[0011] If the updated column paging index is not found, the column paging data corresponding to the updated column paging index is retained to obtain an updated column storage file;

[0012] The updated column storage file is stored in the column storage unit.

[0013] In a second aspect, this embodiment provides a column-based deduplication device, which includes:

[0014] a data receiving module, configured to receive input data and query a column storage file in a column storage unit to determine updated column data, wherein the column storage unit is configured to store a column storage file including column data and column metadata;

[0015] A data paging module is used to paginate the update column data by column, obtain at least one update column paging data, and determine the update column paging index of each update column paging data;

[0016] An index query module, configured to query each of the updated column paging indexes from a column index unit, wherein the column index unit is configured to store column paging index metadata including a column paging index and a storage location of corresponding column paging data;

[0017] a data deletion module configured to delete the column paging data corresponding to the updated column paging index if the updated column paging index is found, and add the updated column paging index corresponding to the deleted column paging data as a deleted column paging index to the column metadata of the column storage file to obtain an updated column storage file;

[0018] A data retention module is used to retain the column paging data corresponding to the updated column paging index if the updated column paging index is not found, and obtain an updated column storage file;

[0019] The file storage module is used to store the updated column storage file in the column storage unit.

[0020] In a third aspect, this embodiment provides an electronic device, including:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the column-based deduplication method according to any embodiment of the present invention.

[0024] An embodiment of the present invention provides a column-based data deduplication method, apparatus, and device, the method comprising: receiving input data, querying a column-based storage file in a column-based storage unit to determine updated column data, the column-based storage unit being used to store a column-based storage file including column data and column metadata; paging the updated column data by column to obtain at least one updated column paging data, and determining an updated column paging index for each updated column paging data; querying each updated column paging index from a column index unit, the column index unit being used to store column paging index metadata including a column paging index and a storage location of corresponding column paging data; if the updated column paging index is queried, deleting the column paging data corresponding to the updated column paging index, and adding the updated column paging index corresponding to the deleted column paging data as a deleted column paging index to the column metadata of the column-based storage file to obtain an updated column-based storage file; if the updated column paging index is not queried, retaining the column paging data corresponding to the updated column paging index to obtain an updated column-based storage file; and storing the updated column-based storage file in the column-based storage unit. The above technical solution realizes online deduplication. It completes data deduplication in real time through a series of operations such as detection and deduplication for input data. Based on the existing columnar storage file format, deduplication-related configurations and properties are added, and a new set of deduplication columnar storage file structures is defined to realize column-level data deduplication. Targeted deduplication only detects and deletes duplicate data on column data, excluding other interfering data, such as metadata, avoiding the contradiction between the block size of the entire file and IO efficiency, and taking into account the deduplication rate and IO efficiency. It can realize the deletion of multiple copies of the same column, the deletion of the same data in different columns, and the deletion of the same data under different compression or encoding, thereby improving the deduplication rate and reducing the impact of additional storage and throughput.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A flowchart of a column-based data deduplication method provided in Example 1 of the present invention;

[0028] Figure 2 This is a diagram illustrating an example structure of deduplicating column-based storage files in the execution of a column-based data deduplication method provided in the first embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a flow chart of another column-based deduplication method provided in the second embodiment of the present invention;

[0030] Figure 4 This is a diagram illustrating a structure of a column-based data deduplication device in a certain application scenario provided by the second embodiment of the present invention;

[0031] Figure 5 A schematic diagram of the structure of a column-based data deduplication device provided in the third embodiment of the present invention;

[0032] Figure 6 This is a structural diagram of an electronic device provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] With the advent of the big data era, more and more data is flowing into the Hadoop ecosystem. At the same time, being able to quickly obtain valuable data from TB or even PB-level data has become more important for a product and company. With the rapid development of the Hadoop ecosystem, open data lake storage formats such as Hudi have emerged. Through high-performance columnar storage formats, they provide massive data OLAP analysis capabilities to meet the increasing demand for high-efficiency data analysis.

[0036] Research shows that up to 60% of the data stored in application systems is redundant, and the proportion of redundant data increases over time. Ensuring reliable and durable data storage can require more than 10 times the storage space and management costs. Consequently, the issue of high data redundancy in storage systems has attracted increasing attention from researchers, and reducing the data storage capacity of storage systems has become a hot research topic. Data deduplication is one such capacity optimization technology. By eliminating duplicate data in storage systems, it reduces the amount of data actually stored in the system or transmitted over the network. It has been widely used in backup, long-term archiving, and data disaster recovery.

[0037] The following describes four commonly used deduplication technologies in existing technologies:

[0038] 1) Full file detection technology. This method uses file-level granularity to find duplicate data. First, a hash is calculated for the entire file. This hash is then compared with the stored hash value. If an identical value is detected, the file is simply replaced with a pointer without actually storing the file. Otherwise, a new file is stored.

[0039] 2) Block detection technology based on the Fixed-Sized Partition (FSP) algorithm. This method uses a block-level, fine-grained approach to detect duplicate data. The FSP algorithm consists of three steps: 1. A predefined block size is provided, and all files are partitioned according to this fixed block size; 2. Each partitioned data block is hashed to obtain a fingerprint value; 3. This value is compared with the stored block fingerprint value. If a match is detected, the data block represented is deleted; otherwise, a new data block is stored.

[0040] 3) Content-Defined Chunking (CDC) detection technology. Unlike fixed chunking strategies, variable chunking detection technology based on the CDC algorithm divides files into chunks of varying lengths. This process involves two steps: First, starting from the file header, a fixed-size sliding window is used to calculate the Rabin fingerprint of the windowed data. The windows overlap, and when the fingerprint value modulo 0 with respect to a specified integer is used, the window position is used as the chunk boundary. A hash algorithm is used to obtain a fingerprint value for each chunk, which is then compared with the stored data chunk. If the fingerprint value is identical, the corresponding chunk is deleted; otherwise, a new chunk is stored.

[0041] 4) Sliding Block Detection Technology. The identical block verification process based on sliding block technology consists of four steps: using a fixed-size sliding window to calculate the sum checksum of each overlapping window in the file; matching the sum checksum with the previously stored value; if a match is found, using the SHA1 algorithm (SHA1 is a hash algorithm) to calculate the hash value, which is further compared with the previously stored value to detect redundancy. If redundancy is found, the sum checksum and hash values ​​of all overlapping windows that did not previously match are recorded, and the current window is skipped; if a match is not found, the sliding window continues to slide.

[0042] The existing technology has the following defects:

[0043] 1) Existing technologies cannot detect redundancy in scenarios where the data content is the same but the columnar storage file formats are different (such as Parquet and ORC, which are two different columnar storage file formats);

[0044] 2) In scenarios where the data content is the same but the serialization encoding methods are different (such as PLAIN natural encoding and INCREMENTAL incremental encoding, PLAIN and INCREMENTAL are two encoding methods), existing technologies cannot detect redundancy.

[0045] 3) In scenarios where the data content is the same but the compression algorithms are different (e.g., SNAPPY and GZIP, SNAPPY and GZIP are two compression algorithms), existing technologies cannot detect redundancy.

[0046] 4) In scenarios where columnar storage format files have the same content but different number of rows grouped together, the deduplication rate of existing technologies is significantly reduced.

[0047] 5) In scenarios where columnar storage format files have the same content but different column paging sizes, the deduplication rate of existing technologies will be significantly reduced;

[0048] 6) The existing technology is a general deduplication technology that lacks targeted optimization for columnar storage format files. It is difficult to balance high deduplication rate and high input / output (I / O) efficiency. If the block is too large, the deduplication rate will be too low. If the block is too small, although the deduplication rate can be improved to a certain extent, file fragmentation will significantly reduce I / O efficiency.

[0049] Therefore, a method for deleting columnar duplicate data is needed to solve the above problems.

[0050] In order to more clearly describe the embodiments of the present invention, some terms are described below. Column storage: Compared with the row storage commonly used in relational databases, all data in each column of the column storage are stored in sequence. There is no need to scan all the data when performing statistical aggregation by column. Therefore, it is widely used in big data processing and storage scenarios, providing high-performance data analysis capabilities. Because of the homogeneity of its column data, it supports efficient encoding and compression, further reducing I / O. Parquet: A column storage format. The file is first divided into row groups (RowGroup) by row, and the RowGroup is divided into column blocks (ColumnChunk) by column. The ColumnChunk is divided into page (Page) storage (default 1MB). ORC: A column storage format. The file is first divided into partition units (Stripe) by row, and the Stripe is stored by column (Column). Index data and statistical information are generated every 10,000 rows by default. Hudi: An open data lake table format that stores data in a column storage format (Parquet / ORC) and provides the ability to update data, delete data, and consume changed data on the Hadoop file system. Deduplication rate: A core metric for deduplication, evaluating the system's ability to identify duplicate data. Rabin fingerprint: A fingerprint algorithm with high computational efficiency and uniform distribution of results for any data. It can quickly calculate the fingerprint of the next window using the fingerprint of the previous window.

[0051] Example 1

[0052] Figure 1This is a flow chart of a column-based deduplication method provided in Example 1 of the present invention. The method is applicable to situations where column-based duplicate data is deleted. The method can be executed by a column-based deduplication device. The column-based deduplication device can be implemented in the form of hardware and / or software and is generally integrated into an electronic device.

[0053] like Figure 1 As shown, the column-based data deduplication method provided in this embodiment may specifically include the following steps:

[0054] S101: Receive input data, and query a column storage file in a column storage unit to determine updated column data.

[0055] In this embodiment, the column storage unit is used to store column storage files that include column data and column metadata. A column storage file consists of each column of data and the corresponding column metadata. Column data refers to the data stored in the column storage file, and column metadata describes the column data. Input data can be specifically understood as data that requires column storage.

[0056] Specifically, the input data to be stored in columnar format is received, and the input data includes a file name or unique identifier, a row, and a column, etc. Then, the column storage unit is queried to determine whether a column storage file corresponding to the input data is stored in the column storage unit. If a column storage file corresponding to the input data does not exist in the column storage unit, a new column storage file is created. If a column storage file corresponding to the input data exists in the column storage unit, the input data needs to be updated to the column storage file. Updating the input data to the column storage file includes different situations, such as adding the input data to the column storage file, or modifying the existing data in the column storage file based on the input data.

[0057] Continuing with the above description, when creating a new column-based storage file or adding data to an existing column-based storage file, the input data is represented as columns to obtain the updated column data. When modifying data in an existing column-based storage file, the input data is modified to the corresponding position in the column-based storage file to obtain the updated column data.

[0058] S102 , paging the update column data by column to obtain at least one update column paging data, and determining an update column paging index for each update column paging data.

[0059] In this embodiment, the updated column data is paginated by column, and the obtained column paging data is recorded as updated column paging data. If the update method of the updated column data is to create a new column storage file or add column data to the column storage file, the updated column data is paginated, and Rabin fingerprints are used for variable paging division. The target page size and fingerprint sliding window are configurable, and multiple column paging data are obtained, which are recorded as updated column paging data. If the update method of the updated column data is to modify the existing data in the column storage file, which will cause the adjacent column paging data to change, Rabin fingerprints are used again for paging division, and the column data associated with the updated column data is paging processed to obtain multiple updated column paging data. Hash calculation is performed on each updated column paging data, and the hash value obtained is used as the index of each column paging data, which is recorded as the updated column paging index.

[0060] S103: Query the paging index of each update column from the column index unit.

[0061] In this embodiment, the column index unit is used to store column paging index metadata including the column paging index and the storage location of the corresponding column paging data. The column paging index can be specifically understood as the index corresponding to the column paging data. The column paging index and the storage location of the corresponding column paging data are collectively recorded as column paging index metadata.

[0062] Specifically, according to the updated column paging index corresponding to the updated column paging data, the column index unit is queried to perform column duplicate data detection.

[0063] S104. If an update column paging index is found, the column paging data corresponding to the update column paging index is deleted, and the update column paging index corresponding to the deleted column paging data is used as a delete column paging index and added to the column metadata of the column storage file to obtain an updated column storage file.

[0064] Specifically, if the update column paging index detection matches the column paging index in the column index unit, then the column storage file reference in the column paging index that matches the column index unit is added. If the number of file references in the index is 2, then the column paging data corresponding to the update column paging index is deleted. The update column paging index corresponding to the deleted column paging data is used as the deletion column paging index and added to the column metadata of the column storage file. In this way, it can be ensured that after obtaining the column storage file, whether the column storage file corresponds to the deleted duplicate data is determined based on whether the column metadata in the column storage file includes the deletion column paging index. Delete the column paging data of the corresponding file referenced by each file in the index, synchronously update the column metadata, and add the column paging index to the column metadata as a reference to the deduplicated data to achieve file formatting after data deduplication.

[0065] Following the above description, the column data after deduplication is serialized and compressed to obtain processed column data, and the processed column data and updated column metadata are stored as an updated column storage file.

[0066] S105: If the updated column paging index is not found, the column paging data corresponding to the updated column paging index is retained to obtain an updated column storage file.

[0067] Specifically, if the updated column paging index does not detect a matching column paging index in the column index unit, the column paging data corresponding to the updated column paging index is retained. The column paging data is serialized and compressed to obtain processed column data, and the processed column data and column metadata are stored as an updated column storage file.

[0068] S106: Store the column storage file in the column storage unit.

[0069] Specifically, the updated column storage file data is submitted to the column storage unit for persistent storage.

[0070] The above technical solution realizes online deduplication. For input data, it completes data deduplication in real time through a series of operations such as detection and deduplication. Based on the existing columnar storage file format, deduplication-related configurations and properties are added, and a new set of deduplication columnar storage file structures is defined to realize column-level data deduplication. Targeted deduplication only detects and deletes duplicate data on column data, excluding other interfering data, such as metadata, avoiding the contradiction between the entire file block size and I / O efficiency, and taking into account the deduplication rate and I / O efficiency. It can realize the deletion of multiple copies of the same column, the deletion of the same data in different columns, and the deletion of the same data under different compression or encoding, thereby improving the deduplication rate and reducing the impact of additional storage and throughput.

[0071] As an optional embodiment of the present invention, based on the above embodiment, after deleting the column paging data corresponding to the column paging index, the method may be optimized to further include:

[0072] a1) The deleted column paging index and the storage location of the column paging data corresponding to the deleted column paging index are stored in the column index unit as column paging index metadata.

[0073] Specifically, after deleting the column paging index corresponding to the column paging index, the deleted column paging index and the storage location of the column paging data corresponding to the deleted column paging index need to be stored in the column index unit as column paging index metadata. It should be noted that if the column storage file corresponds to duplicate deleted data, the column metadata of the column storage file will include the deleted column paging index. By querying the column index unit, the storage location of the deleted column paging index and the corresponding deleted data in the deduplication data unit can be found, and then the deduplicated column paging data can be obtained based on the storage location.

[0074] b1) Storing the deleted column paging data in the deduplication data unit.

[0075] In this embodiment, the deduplication data unit is used to store deleted column page data. The deleted column page data is serialized and compressed uniformly and then stored in the deduplication data unit.

[0076] In order to more clearly illustrate the column-based data deduplication method provided by the embodiment of the present invention, the structure of the deduplication column-based storage file is described. Figure 2 This is a structural example diagram of deduplicating column-type storage files in a column-type deduplication method according to the first embodiment of the present invention, as shown in FIG. Figure 2 As shown, the column storage file consists of each column data and its corresponding column metadata. The column data uses the Rabin fingerprint as the split point, sets a sliding window of configurable size, slides to calculate the Rabin fingerprint, modulo the Rabin fingerprint value with the target page size (such as 8KB, configurable), and if it is 0, it is used as the split point, and the Rabin fingerprint is inserted into the data, and the split point position is recorded in the column metadata. For duplicate pages detected in the column data, after deletion, the column page index (shown as the page hash in the figure) is recorded in the column metadata, and the order of the duplicated data position is maintained. If the Rabin fingerprint is invalid due to modification of the column page data, the corresponding Rabin fingerprint is deleted, the fingerprint is recalculated, and the hash index corresponding to the adjacent column page is updated. Modifying the data will only affect the nearby column pages and will not cause changes to other pages. The deduplicated page hash in the column metadata points to the page hash column in the column index. The deduplicated data address is used to query the deduplicated data storage, such as the Hadoop Distributed File System (HDFS) or object storage (configurable), to obtain the deduplicated column page data. The figure shows that there is a deduplicated page between the column pages represented by Rabin fingerprint_1 and Rabin fingerprint_2. This page data is retrieved from the deduplicated data unit and merged into the column data when the column data is output, forming the complete column data.

[0077] It can be understood that by defining multiple components such as column data, column metadata, column paging, sliding windows, Rabin fingerprints, column paging indexes, and deduplicated data storage, the separation of columnar storage files and deduplicated data and variable paging of column data are achieved, which has the characteristics of high deduplication rate, high I / O efficiency, and resistance to data displacement.

[0078] The above technical solution adds the storage of the deleted column paging index in the column index unit and the storage of the deleted data in the deduplication data unit, which can support column paging index storage and query, and can obtain deduplication data reference information and deduplication column storage file reference information based on the column paging index, thereby realizing the maintenance of the relationship between deduplication data and column storage files.

[0079] As an optional embodiment of the present invention, based on the above embodiment, after retaining the column paging data corresponding to the column paging index, the following steps may be further optimized:

[0080] The updated column paging index and the storage location of the column paging data corresponding to the updated column paging index are stored in the column index unit as column paging index metadata.

[0081] Specifically, an update column paging index is added to the column index unit. The index records the column paging index and a reference to the column storage file that references the column paging index. The column storage file can be marked with a file name or a unique identifier. The update column paging index and the storage location of the column paging data corresponding to the update column paging index are stored in the column index unit as column paging index metadata.

[0082] The above technical solution adds a function of storing the updated column paging index and the storage location of the column paging data corresponding to the updated column paging index in the column index unit, which provides a basis for subsequent column data query.

[0083] As an optional embodiment of the embodiment of the present invention, based on the above embodiment, the method further includes:

[0084] Obtain output column data according to the column storage file in the column storage unit.

[0085] The above describes the process of how to delete duplicate data in the input data. The following describes the process of how to merge the deleted duplicate data into the column data of the column storage file to obtain complete column data. In this embodiment, the output column data can be specifically understood as the inverse process of the input data, that is, restoring the column storage file from which duplicate data has been deleted back to the original complete data. Specifically, in order to restore the column storage file in the column storage unit back to the output column data, it is necessary to combine the column index unit and the deduplication data unit to restore the output column data.

[0086] As a specific implementation, the step of obtaining output column data according to the column storage file in the column storage unit can be optimized, including:

[0087] a2) Reading a column storage file from the column storage unit to determine original column data contained in the column storage file.

[0088] In this embodiment, the column storage file format of a column storage file is identified; based on the column storage file format, column metadata of the column storage file is read to obtain a compression algorithm and a serialization algorithm; based on the compression algorithm, the column data in the column storage file is decompressed to obtain decompressed column data; based on the serialization algorithm, the decompressed column data is deserialized to obtain the original column data corresponding to the column storage file. A read operation is performed on the decompressed and deserialized original column data to read the paging data of each column.

[0089] In this embodiment, a column storage file is read from a column storage unit, and a format recognition is performed on the column storage file to obtain a file format of the column storage file, which is recorded as the column storage file format. Exemplarily, the column storage file format can be Parquet, ORC, or other formats.

[0090] It's understandable that different column-based storage file formats use different methods for data storage and record column metadata in different ways. Therefore, depending on the specific column-based storage file format, it's necessary to read the column metadata in the column-based storage file from the corresponding location to determine the compression and serialization algorithms used to store the column data in the column-based storage file. For example, compression algorithms might include SNAPPY and GZIP, and serialization algorithms might include PLAIN encoding and INCREMENTAL encoding.

[0091] In this embodiment, after obtaining the compression algorithm used to compress the column data, a corresponding decompression algorithm can be used based on the compression algorithm to decompress the column data in the column storage file, thereby obtaining the decompressed column data. For example, assuming that the compression algorithm obtained is SNAPPY, the decompression algorithm corresponding to SNAPPY is used to decompress the column data in the column storage file to obtain the decompressed column data. assuming that the compression algorithm obtained is GZIP, the decompression algorithm corresponding to GZIP is used to decompress the column data in the column storage file to obtain the decompressed column data.

[0092] In this embodiment, the original column data can be specifically understood as the original input data before storage in a columnar storage manner. After obtaining the decompressed column data, it is also necessary to perform deserialization processing on it, so as to obtain the data after serialization processing, that is, the original column data corresponding to the columnar storage file. Exemplarily, assuming that the serialization algorithm obtained is PLAIN natural encoding, the deserialization algorithm corresponding to PLAIN natural encoding is used to deserialize the decompressed column data to obtain the deserialized original column data. Assuming that the serialization algorithm obtained is INCREMENTAL incremental encoding, the deserialization algorithm corresponding to INCREMENTAL incremental encoding is used to deserialize the decompressed column data to obtain the deserialized original column data. Subsequently, data is added to the decompressed and deserialized original column data or existing data is modified.

[0093] b2) Parse the column metadata in the column storage file to determine whether the column metadata includes a deleted column paging index.

[0094] Specifically, the column metadata in the column storage file is read and parsed, the column paging index recorded in the column metadata is obtained, and whether the column metadata contains a deletion paging index is determined. If the column metadata contains a deletion paging index, it indicates that the column data in the column storage file does not contain the complete input data. The deleted duplicate data must be obtained from the deduplication data unit and merged to restore the input data. If the column metadata does not contain a deletion paging index, it indicates that the column data in the column storage file contains the complete input data, and no further data needs to be obtained from the deduplication data unit.

[0095] c2) If the deleted column paging index is included, query the column index unit to determine the storage location of the column paging data corresponding to the deleted column paging index in the deduplication data unit.

[0096] Specifically, if the column metadata contains a deleted column paging index, the column index unit is queried. The column index unit contains column paging index metadata, which records the column paging index and the storage location of the corresponding column paging data. Therefore, the column index unit can be queried to query the deleted column paging index, thereby determining the storage location of the column paging data corresponding to the deleted column paging index in the deduplication data unit.

[0097] d2) According to the storage location, query the deduplication data unit to obtain the deduplication column paging data corresponding to the column storage file.

[0098] Specifically, based on the information of the storage location of the deduplication data, the deduplication data unit is queried to obtain the data at this storage location, which is the deduplication column paging data corresponding to the column storage file.

[0099] e2) Merge the duplicate removal column paging data with the original column data to obtain output column data.

[0100] It is understandable that the column paging index contained in the column index unit records the placeholder of its corresponding data, so the duplicated column paging data can be merged into the column paging data in the original column data according to its placeholder to form complete column data.

[0101] f2) If the paging index of the deleted column is not included, the original column data is used as the output column data.

[0102] Specifically, if the column metadata does not include a delete paging index, it means that the column data in the column storage file contains complete input data, and no additional data needs to be obtained from the deduplication data unit. That is, the original column data is used as output column data.

[0103] The above technical solution specifies the steps for obtaining output column data from column-based storage files in a column-based storage unit. This eliminates the complexity of reading deduplicated column-based storage files, is transparent to users, and automatically merges and dedupes file data, providing out-of-the-box column-based data reading capabilities.

[0104] Example 2

[0105] Figure 3 A flow chart of another column-based deduplication method provided in the second embodiment of the present invention. This embodiment is a further optimization of the above embodiment. In this embodiment, the optimization is further limited to "receiving input data, querying the column-based storage file in the column-based storage unit to determine the updated column data", and the optimization is further limited to "paging the updated column data by column, obtaining at least one updated column paging data, and determining the updated column paging index of each of the updated column paging data".

[0106] like Figure 3 As shown, this embodiment 2 provides a column-based deduplication method, which specifically includes the following steps:

[0107] S201: Receive input data and query whether there is a column storage file corresponding to the input data in the column storage unit.

[0108] Specifically, input data to be stored in columnar format is received, and the input data includes a file name or unique identifier, a row, and a column, etc. Then, the column storage unit is queried to determine whether a column storage file corresponding to the input data is stored in the column storage unit. Exemplarily, the judgment method may be to compare the file name or unique identifier of the input data with the file name or unique identifier of each column storage file in the column storage unit. If there is a column storage file that is identical to the file name or unique identifier of the input data, it indicates that there is a column storage file corresponding to the input data in the column storage unit; if there is no column storage file that is identical to the file name or unique identifier of the input data, it indicates that there is no column storage file corresponding to the input data in the column storage unit.

[0109] S202: If a corresponding column storage file does not exist, create a new column storage file and use the input data as update column data.

[0110] Specifically, if a column storage file corresponding to the input data does not exist in the column storage unit, a new column storage file is created, and the input data is used as the updated column data.

[0111] S203: If a corresponding column storage file exists, determine the original column data corresponding to the column storage file, and add column data or modify existing column data to the original column data according to the input data to obtain updated column data.

[0112] In this embodiment, if there is a column storage file corresponding to the input data in the column storage unit, the input data needs to be updated to the column storage file. Updating the input data to the column storage file includes different situations, such as adding the input data to the column storage file, or modifying the existing data in the column storage file based on the input data. Specifically, if there is a column storage file corresponding to the input data, the column storage file format is identified, the column data is decompressed and deserialized, and the original column data corresponding to the column storage file is obtained. Then, according to the input data, the original column data is processed to add column data or modify the existing column data to obtain the updated column data.

[0113] As a specific implementation method, the steps of determining the original column data corresponding to the column storage file can be optimized, including:

[0114] a3) Identify the column storage file format of the column storage file.

[0115] In this embodiment, the format of the column storage file is identified to obtain the file format of the column storage file, which is recorded as the column storage file format. Exemplarily, the column storage file format can be Parquet, ORC, etc.

[0116] b3) Read the column metadata of the column storage file according to the column storage file format to obtain the compression algorithm and serialization algorithm.

[0117] It's understandable that different column-based storage file formats use different methods for data storage and record column metadata in different ways. Therefore, depending on the specific column-based storage file format, it's necessary to read the column metadata in the column-based storage file from the corresponding location to determine the compression and serialization algorithms used to store the column data in the column-based storage file. For example, compression algorithms might include SNAPPY and GZIP, and serialization algorithms might include PLAIN encoding and INCREMENTAL encoding.

[0118] c3) Decompressing the column data in the column storage file according to the compression algorithm to obtain decompressed column data.

[0119] In this embodiment, after obtaining the compression algorithm used to compress the column data, a corresponding decompression algorithm can be used based on the compression algorithm to decompress the column data in the column storage file, thereby obtaining the decompressed column data. For example, assuming that the compression algorithm obtained is SNAPPY, the decompression algorithm corresponding to SNAPPY is used to decompress the column data in the column storage file to obtain the decompressed column data. assuming that the compression algorithm obtained is GZIP, the decompression algorithm corresponding to GZIP is used to decompress the column data in the column storage file to obtain the decompressed column data.

[0120] d3) Deserialize the decompressed column data according to the serialization algorithm to obtain the original column data corresponding to the column storage file.

[0121] In this embodiment, the original column data can be specifically understood as the original input data before storage in a columnar storage manner. After obtaining the decompressed column data, it is also necessary to perform deserialization processing on it, so as to obtain the data after serialization processing, that is, the original column data corresponding to the columnar storage file. Exemplarily, assuming that the serialization algorithm obtained is PLAIN natural encoding, the deserialization algorithm corresponding to PLAIN natural encoding is used to deserialize the decompressed column data to obtain the deserialized original column data. Assuming that the serialization algorithm obtained is INCREMENTAL incremental encoding, the deserialization algorithm corresponding to INCREMENTAL incremental encoding is used to deserialize the decompressed column data to obtain the deserialized original column data. Subsequently, data is added to the decompressed and deserialized original column data or existing data is modified.

[0122] The above technical solution specifies the steps for determining the original column data corresponding to a columnar storage file. This is achieved through columnar storage file format recognition, decompression, and deserialization. The columnar storage file recognition process shields the differences between various columnar storage file types in the industry, providing a wide range of adaptability. The decompression and deserialization processes shield the same column data with different compression and serialization algorithms, avoiding the problem of duplicate data being unable to be deleted and improving the deduplication rate.

[0123] S204: If the update mode of the updated column data is to create a new column storage file or to add column data to the column storage file, a set paging algorithm is used to perform paging processing on the updated column data to obtain at least one updated column paging data.

[0124] The paging algorithm can use Rabin fingerprints. Specifically, if the updated column data is updated by creating a new column storage file or adding column data to a column storage file, the updated column data is paginated using Rabin fingerprints for variable paging. The target page size and fingerprint sliding window are configurable, resulting in multiple column paging data, which are recorded as updated column paging data.

[0125] S205 : If the updating method of the updated column data is to modify existing data in the column storage file, a set paging algorithm is used to perform paging processing on the column data associated with the updated column data to obtain at least one updated column paging data.

[0126] Specifically, if the update method of the updated column data is to modify the existing data in the column storage file, which will cause the adjacent column paging data to change, the Rabin fingerprint is reused for paging, and the column data associated with the updated column data is paging processed to obtain multiple updated column paging data.

[0127] It's important to note that by re-paging columnar data, we mask differences in row grouping strategies across various columnar storage files, such as the default 1MB for Parquet and 10,000 rows for ORC, improving the deduplication rate. Rabin fingerprint variable paging prevents data displacement caused by data modifications, which could invalidate the original paging and support data insertion and updates.

[0128] S206 , performing hash calculation on each updated column paging data, and using the calculation result as the updated column paging index of each column paging data.

[0129] Specifically, a hash calculation is performed on each updated column paging data to obtain a hash value as an index of each column paging data, which is recorded as the updated column paging index. In this embodiment, there is no specific limitation on the hash algorithm. For example, the Message Digest Algorithm Version 5 (MD5) and the Secure Hash Algorithm (SHA1) can be used.

[0130] S207: Query the paging index of each update column from the column index unit.

[0131] The column index unit is used to store column paging index metadata including a column paging index and a storage location of corresponding column paging data.

[0132] S208. If an update column paging index is found, the column paging data corresponding to the update column paging index is deleted, and the update column paging index corresponding to the deleted column paging data is used as a delete column paging index and added to the column metadata of the column storage file to obtain an updated column storage file.

[0133] S209: If the updated column paging index is not found, the column paging data corresponding to the updated column paging index is retained to obtain an updated column storage file.

[0134] S210: Store the updated column storage file in the column storage unit.

[0135] The above technical solution specifies the steps of receiving input data, querying the column storage file in the column storage unit to determine the updated column data; and paging the updated column data by column, obtaining at least one updated column paging data, and determining the updated column paging index of each updated column paging data. For big data processing and storage scenarios, column-level duplicate data detection, encoding, and deletion are used to improve the deduplication rate and save storage space. It supports the deletion of multiple copies of the same column, the deletion of the same data in different columns, and the deletion of the same data under different compression or encoding conditions. It has the characteristics of high deduplication rate, low additional storage, and low throughput impact.

[0136] It should be noted that this technical solution uses an online deduplication method to achieve in-process duplicate data detection and deduplication during the data addition and modification process, with high timeliness. It supports automatic columnar storage file format identification, decompression, and deserialization, and can mask differences other than file content, ensuring that files with the same file content but different file formats, compression algorithms, and serialization algorithms can also correctly detect and delete duplicate data, with a high deduplication rate. It supports storage and query services for various columnar storage files, not limited to columnar storage file formats such as Parquet and ORC. It supports column paging index storage and query, and can obtain deduplication data reference information and deduplication column storage file reference information based on column paging hash, realizing the maintenance of the relationship between deduplication data and column storage files. It supports storage and query of deduplication data, not limited to various storage systems such as HDFS, object storage, and key-value pair (KV) storage. It can automatically identify, decompress, deserialize, and merge column data, mask differences, and is transparent to users. Column data generation and output are completed during queries, with high timeliness. A customized deduplication column storage file structure is adopted. By defining multiple components such as column data, column metadata, column paging, sliding window, Rabin fingerprint, column paging index, and deduplication data storage, the column storage file and deduplication data separation and variable paging of column data are realized. It has the characteristics of high deduplication rate, high IO efficiency, and resistance to data displacement. Data deduplication is performed on column storage data, paging by column, parsing by column, detecting by column, deduplicating by column, and merging by column. Due to the homogeneity of column data, the deduplication rate is high. In addition, the present invention detects and deduplicates the content of column data, eliminating metadata information with low repetition rate and scattered distribution, further improving the deduplication rate and better adapting to the storage and analysis characteristics of the current big data era. It is highly scalable. The compression algorithm, column storage format, paging size, serialization algorithm, deduplication data storage system, and index structure can all be flexibly customized and configured. It is connected through standardized interfaces to adapt to a wide range of application scenarios. The file structure self-analysis is completed by reading the column storage file without manual intervention or customized configuration.

[0137] In order to more clearly illustrate the column-based deduplication method provided by the embodiment of the present invention, an actual application scenario of column-based deduplication is used as an example for illustration. Figure 4 This is a diagram showing an example of the structure of a column-based deduplication device in a certain application scenario provided by the second embodiment of the present invention. Figure 4 As shown in the figure, it is assumed that different functions of the column-based deduplication method are integrated into different units, namely: online deduplication unit U1, column storage unit U2, column index unit U3, deduplication data unit U4 and column reading unit U5. The functions of each unit are introduced below:

[0138] Online deduplication unit U1: Input data. If the column storage file already exists, it is considered a modification. The column storage file is first read from U2 to insert or modify the data. If a new column storage file is to be created, a new column storage file is created. After the column storage file is modified or created, the column data is paged by column. The column paging index is queried from U3. If it can be found, the column paging data is deleted and replaced with column metadata. If it cannot be found, the column paging index is written to U3.

[0139] Column storage unit U2: stores column storage file data, provides column storage file read services to U1 and U5, and provides column storage file write services to U1.

[0140] The column index unit U3 stores column paging index metadata and provides column paging index metadata query services to U1 and U5, including the hash of column paging and the storage location of column paging data in U4, and provides column paging index metadata write services to U1.

[0141] The deduplication data unit U4 stores the deduplication column paging data and provides deduplication column paging data query services to U1 and U4.

[0142] The column-based reading unit U5 first reads the column-based storage file in U2 and parses the column metadata. If it contains deduplicated data, it queries U3 based on the column paging hash in the column metadata. Based on the query results, it further queries U4 to obtain the deduplicated column paging data, merges it into the column-based storage file, and outputs the column data.

[0143] Through the interaction of the above five units, duplicate data in the input data can be deleted, and the deleted duplicate data can be restored to obtain output data.

[0144] Example 3

[0145] Figure 5 This is a schematic diagram of the structure of a column-based deduplication device provided in the third embodiment of the present invention. The device is applicable to the case of deleting column-based duplicate data. The column-based deduplication device can be implemented in the form of hardware and / or software and is generally integrated into an electronic device. Figure 5 As shown, the device includes: a data receiving module 31, a data paging module 32, an index query module 33, a data deletion module 34, a data retention module 35 and a file storage module 36, wherein:

[0146] A data receiving module 31 is configured to receive input data and query a column storage file in a column storage unit to determine updated column data, where the column storage unit is configured to store a column storage file including column data and column metadata;

[0147] The data paging module 32 is used to paging the update column data by column, obtain at least one update column paging data, and determine the update column paging index of each update column paging data;

[0148] An index query module 33, configured to query each updated column paging index from a column index unit, wherein the column index unit is configured to store column paging index metadata including a column paging index and a storage location of corresponding column paging data;

[0149] The data deletion module 34 is configured to delete the column paging data corresponding to the updated column paging index if an updated column paging index is found, and add the updated column paging index corresponding to the deleted column paging data as a deleted column paging index to the column metadata of the column storage file to obtain an updated column storage file;

[0150] The data retention module 35 is configured to retain the column paging data corresponding to the updated column paging index if the updated column paging index is not found, and obtain an updated column storage file;

[0151] The file storage module 36 is configured to store the updated column storage file in the column storage unit.

[0152] The above technical solution implements online deduplication, performing real-time data deduplication on input data through a series of operations, including detection and deduplication. Based on the existing columnar storage file format, deduplication-related configurations and properties are added, and a new deduplication columnar storage file structure is defined to implement column-level data deduplication. Targeted deduplication only detects and removes duplicate data within the column, excluding other interfering data, such as metadata. This avoids the conflict between the overall file block size and I / O efficiency, and balances the deduplication rate and I / O efficiency.

[0153] Optionally, the data receiving module 31 may include:

[0154] A file determination unit is configured to receive input data and query whether a column storage file corresponding to the input data exists in the column storage unit;

[0155] A file creation unit is used to create a new column storage file if the corresponding column storage file does not exist, and use the input data as the updated column data;

[0156] The file modification unit is used to determine the original column data corresponding to the column storage file if a corresponding column storage file exists, and add column data or modify the existing column data to the original column data according to the input data to obtain updated column data.

[0157] Optionally, the file modification unit is specifically configured to:

[0158] Identify the column storage file format of column storage files;

[0159] According to the column storage file format, read the column metadata of the column storage file to obtain the compression algorithm and serialization algorithm;

[0160] Decompress the column data in the column storage file according to the compression algorithm to obtain the decompressed column data;

[0161] The decompressed column data is deserialized according to the serialization algorithm to obtain the original column data corresponding to the column storage file.

[0162] Optionally, the data paging module is specifically used to:

[0163] If the update method of the updated column data is to create a new column storage file or add column data to the column storage file, the set paging algorithm is used to perform paging processing on the updated column data to obtain at least one updated column paging data;

[0164] If the update method of the updated column data is to modify the existing data in the column storage file, the set paging algorithm is used to perform paging processing on the column data associated with the updated column data to obtain at least one updated column paging data;

[0165] Perform hash calculation on each updated column paging data, and use the calculation result as the updated column paging index of each column paging data.

[0166] Optionally, the device further includes a deletion storage module, which is configured to:

[0167] The column paging index and the storage location of the column paging data corresponding to the deleted column paging index are stored in the column index unit as column paging index metadata;

[0168] The deleted column paging data is stored in a deduplication data unit, which is used to store the deleted column paging data.

[0169] Optionally, the device further includes an index storage module, which, after retaining the column paging data corresponding to the column paging index, is used to:

[0170] The updated column paging index and the storage location of the column paging data corresponding to the updated column paging index are stored in the column index unit as column paging index metadata.

[0171] Optionally, the device further includes a data output module, configured to:

[0172] Obtain output column data according to the column storage file in the column storage unit.

[0173] Optionally, the data output module is specifically configured to:

[0174] Read the column storage file from the column storage unit and determine the original column data contained in the column storage file;

[0175] Parse the column metadata in the column storage file to determine whether the column metadata contains the deleted column paging index;

[0176] If the deleted column paging index is included, query the column index unit to determine the storage location of the column paging data corresponding to the deleted column paging index in the deduplication data unit;

[0177] Query the deduplication data unit based on the storage location to obtain the deduplication column paging data corresponding to the column storage file;

[0178] Merge the duplicate removal column paging data with the original column data to obtain output column data;

[0179] If the paging index of the deleted column is not included, the original column data will be used as the output column data.

[0180] The column-based deduplication device provided in the embodiment of the present invention can execute the column-based deduplication method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0181] Example 4

[0182] Figure 6 A schematic diagram of the structure of an electronic device provided for embodiment four of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0183] like Figure 6As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., which is communicatively connected to the at least one processor 41. The memory stores a computer program that can be executed by the at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. Various programs and data required for the operation of the electronic device 40 can also be stored in the RAM 43. The processor 41, ROM 42, and RAM 43 are connected to each other via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0184] Multiple components in the electronic device 40 are connected to the I / O interface 45, including an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0185] Processor 41 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. Processor 41 executes the various methods and processes described above, such as the column-based deduplication method.

[0186] In some embodiments, the column-based data deduplication method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the column-based data deduplication method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the column-based data deduplication method in any other suitable manner (e.g., via firmware).

[0187] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0189] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0192] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0193] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the column-based deduplication method provided in any embodiment of the present invention.

[0194] The computer program product may be implemented by writing computer program code for performing the operations of the present disclosure in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0195] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0196] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A column-based deduplication method, characterized in that: include: receiving input data, and querying a column storage file in a column storage unit to determine updated column data, wherein the column storage unit is used to store a column storage file including column data and column metadata; Paging the update column data by column to obtain at least one update column paging data, and determining the update column paging index of each update column paging data; querying each of the updated column paging indexes from a column index unit, wherein the column index unit is used to store column paging index metadata including a column paging index and a storage location of corresponding column paging data; If the update column paging index is found, the column paging data corresponding to the update column paging index is deleted, and the update column paging index corresponding to the deleted column paging data is used as the deletion column paging index and added to the column metadata of the column storage file to obtain the updated column storage file; If the updated column paging index is not found, the column paging data corresponding to the updated column paging index is retained to obtain an updated column storage file; The updated column storage file is stored in the column storage unit.

2. The method according to claim 1, characterized in that The receiving input data and querying the column storage file in the column storage unit to determine the updated column data includes: Receive input data, and query whether there is a column storage file corresponding to the input data in the column storage unit; If the corresponding column storage file does not exist, create a new column storage file and use the input data as the updated column data; If a corresponding column storage file exists, original column data corresponding to the column storage file is determined, and column data is added or existing column data is modified on the original column data according to the input data to obtain the updated column data.

3. The method according to claim 2, characterized in that The determining the original column data corresponding to the column storage file includes: Identifying a column storage file format of the column storage file; According to the column storage file format, read the column metadata of the column storage file to obtain a compression algorithm and a serialization algorithm; Decompressing the column data in the column storage file according to the compression algorithm to obtain decompressed column data; The decompressed column data is deserialized according to the serialization algorithm to obtain the original column data corresponding to the column storage file.

4. The method according to claim 1, wherein The step of paging the update column data by column to obtain at least one update column paging data and determining the update column paging index of each update column paging data includes: If the updating method of the updated column data is to create a new column storage file or add column data to the column storage file, a set paging algorithm is used to perform paging processing on the updated column data to obtain at least one updated column paging data; If the update mode of the update column data is to modify the existing data in the column storage file, a set paging algorithm is used to perform paging processing on the column data associated with the update column data to obtain at least one update column paging data; A hash calculation is performed on each of the updated column paging data, and the calculation result is used as the updated column paging index of each of the column paging data.

5. The method according to claim 1, wherein After deleting the column paging data corresponding to the column paging index, the method further includes: The deleted column paging index and the storage location of the column paging data corresponding to the deleted column paging index are stored in the column index unit as column paging index metadata; The deleted column paging data is stored in a deduplication data unit, where the deduplication data unit is used to store the deleted column paging data.

6. The method according to claim 1, wherein After retaining the column paging data corresponding to the column paging index, the method further includes: The updated column paging index and the storage location of the column paging data corresponding to the updated column paging index are stored in the column index unit as column paging index metadata.

7. The method according to claim 1, characterized in that Also includes: Output column data is obtained according to the column storage file in the column storage unit.

8. The method according to claim 7, characterized in that The step of obtaining output column data according to the column storage file in the column storage unit includes: Reading a column storage file from the column storage unit, and determining original column data contained in the column storage file; Parsing the column metadata in the column storage file to determine whether the column metadata includes a deleted column paging index; If the deleted column paging index is included, query the column index unit to determine the storage location of the column paging data corresponding to the deleted column paging index in the deduplication data unit; According to the storage location, query the deduplication data unit to obtain the deduplication column paging data corresponding to the column storage file; Merging the deduplication column paging data with the original column data to obtain the output column data; If the paging index of the deleted column is not included, the original column data is used as the output column data.

9. A column-based deduplication device, characterized in that: include: a data receiving module, configured to receive input data and query a column storage file in a column storage unit to determine updated column data, wherein the column storage unit is configured to store a column storage file including column data and column metadata; A data paging module is used to paginate the update column data by column, obtain at least one update column paging data, and determine the update column paging index of each update column paging data; An index query module, configured to query each of the updated column paging indexes from a column index unit, wherein the column index unit is configured to store column paging index metadata including a column paging index and a storage location of corresponding column paging data; a data deletion module configured to delete the column paging data corresponding to the updated column paging index if the updated column paging index is found, and add the updated column paging index corresponding to the deleted column paging data as a deleted column paging index to the column metadata of the column storage file to obtain an updated column storage file; A data retention module is used to retain the column paging data corresponding to the updated column paging index if the updated column paging index is not found, and obtain an updated column storage file; The file storage module is used to store the updated column storage file in the column storage unit.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the column-based deduplication method according to any one of claims 1 to 8.

Images