Data deduplication method, device, terminal and storage medium
The data is deduplicated through compressed bitmap technology, which solves the IO redundancy and resource waste problems of duplicate data processing in real-time data synchronization scenarios, and achieves rapid deduplication and system stability improvement.
Patent Information
- Application Number
- CN202410917323.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-10
AI Technical Summary
When the prior art processes duplicate data in real-time data synchronization scenarios, there are problems of IO redundancy consumption, memory and CPU computing resources, external storage resources and system instability.
Using compressed bitmap technology, by splicing the comparison field data and encoding it according to preset rules, it quickly identifies duplicate data, reduces memory usage and database writing pressure, and avoids external storage dependencies.
It effectively reduces memory and CPU resource consumption, reduces database write pressure, improves system stability, and realizes rapid deduplication processing.
Smart Images

Figure CN118467590B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data deduplication method, device, terminal and storage medium. Background Art
[0002] In the real-time data synchronization scenario, when duplicate data appears, the solutions and problems of handling duplicate data in the prior art are as follows:
[0003] 1. Deduplication is performed by downstream data storage (such as database).
[0004] Existing problems: Duplicate data will cause redundant IO consumption in data transmission, and the downstream database will write or update deduplication based on the database's index mechanism, increasing the write pressure of the downstream database.
[0005] 2. Perform cache grouping deduplication in the synchronous stream.
[0006] Existing problems: When performing deduplication comparison, all field data to be compared must be cached (that is, multiple rows and columns of data must be cached for comparison). A large amount of data needs to be stored in the memory, and the deduplication operation consumes a large amount of memory and CPU computing resources.
[0007] 3. Use external storage for deduplication during synchronization.
[0008] Existing problems: Using external data such as Redis or Hbase to store the data to be compared will increase the complexity of system interaction and cause instability of real-time tasks. It will also increase the waste of external storage resources. Summary of the invention
[0009] In view of this, the embodiments of the present application provide a data deduplication method, device, terminal and storage medium, which can realize rapid deduplication of data.
[0010] In a first aspect, an embodiment of the present application provides a data deduplication method, comprising:
[0011] Obtain a target data table corresponding to the deduplication task, each of the target data tables being configured with at least one compressed bitmap; wherein the compressed bitmap comprises a plurality of bits having a preset number of bits, and each of the bits is provided with a corresponding state identifier;
[0012] Determine the comparison field data of each target data table in each deduplication task;
[0013] splicing all comparison field data in the same record unit of each target data table to form a splicing character, and encoding the splicing character according to the preset number of bits to obtain the comparison value corresponding to each record unit respectively;
[0014] Detecting respectively whether there are bits in each of the compressed bitmaps that match the comparison value;
[0015] When the corresponding bit position is matched for the first time, the state flag of the corresponding bit position in the matched current compression bitmap is changed from the first preset value to the second preset value;
[0016] When the corresponding bit position is matched again, if the state identifier of the corresponding bit position in any one of the compressed bitmaps is the second preset value, it is determined that the record unit corresponding to the current comparison value is repeated data, and the repeated data is output.
[0017] In some embodiments, each of the target data tables is configured with at least two of the compressed bitmaps;
[0018] After respectively detecting whether there are bits in each of the compressed bitmaps that match the comparison value, the method further includes:
[0019] When the corresponding bit positions are matched, if the state flags of the corresponding bit positions are all first preset values, determining whether a preset scrolling time of the currently written compressed bitmap has been reached;
[0020] If so, the current comparison value is written into the corresponding bit in the next compression bitmap corresponding to the current target data table, and the state flag of the current bit is changed from the first preset value to the second preset value.
[0021] In some embodiments, the method further comprises: when the preset scrolling time of the last compressed bitmap corresponding to the current target data table is reached, clearing the data in the bit of the first compressed bitmap corresponding to the current target data table, and changing all the second preset values of the first compressed bitmap to the first preset values;
[0022] The current first compressed bitmap is used as the last compressed bitmap, and the second compressed bitmap corresponding to the current target data table is used as the first compressed bitmap. The remaining compressed bitmaps are sorted in sequence, and the data in the current target data table is continued to be deduplicated until all the data in the current target data table is deduplicated.
[0023] In some embodiments, before determining the comparison field data of each target data table in each deduplication task, the method further includes:
[0024] The comparison fields configured in each target data table are obtained, and all the comparison fields in the same target data table are saved with identification marks.
[0025] In some embodiments, determining the comparison field data of each target data table in each deduplication task includes:
[0026] According to the identification mark of each target data table in each deduplication task, obtaining the comparison field of each target data table from all the comparison fields with the identification mark saved;
[0027] According to the comparison field of each target data table, comparison field data is extracted from the corresponding target data table.
[0028] In some embodiments, the preset number of bits of comparison value is a 32-bit comparison value or a 64-bit comparison value.
[0029] In some embodiments, the second preset value is 1; and the first preset value is 0.
[0030] In a second aspect, an embodiment of the present application provides a data deduplication device, comprising:
[0031] An acquisition module, used for acquiring a target data table corresponding to a deduplication task, each of the target data tables being configured with at least one compressed bitmap; wherein the compressed bitmap comprises a plurality of bits having a preset number of bits, and each of the bits is provided with a corresponding state identifier;
[0032] A determination module, used to determine the comparison field data of each target data table in each deduplication task;
[0033] An encoding module, used for splicing all comparison field data in the same record unit of each target data table to form a splicing character, and encoding the splicing character according to the preset number of bits to obtain the comparison value corresponding to each record unit respectively;
[0034] A detection module, used to detect whether there are bits in the compressed bitmap that match the comparison value;
[0035] A changing module, used for changing the state flag of the corresponding bit from a first preset value to a second preset value when the corresponding bit is matched for the first time;
[0036] An output module is used to determine that the record unit corresponding to the current comparison value is duplicate data and output the duplicate data when the corresponding bit is matched again if the state identifier of the corresponding bit in any of the compressed bitmaps is the second preset value.
[0037] In a third aspect, an embodiment of the present application provides a terminal device, which includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned data deduplication method.
[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the above-mentioned data deduplication method when executed on a processor.
[0039] The embodiments of the present application have the following beneficial effects: the present application forms a preset comparison value by splicing the comparison field data corresponding to the comparison field and encoding it according to a preset rule, and then searches for data in the corresponding multiple compressed bitmaps corresponding to the comparison value. If so, it indicates that the record corresponding to the comparison value is duplicate data. That is, the present application uses a compressed bitmap to perform data deduplication processing, splicing the comparison field data, encoding it according to a preset rule, and then quickly determining whether there is duplicate data within a period of time through the encoded comparison value. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A first flow chart of the data deduplication method according to an embodiment of the present application is shown;
[0042] Figure 2 A second flow chart of the data deduplication method according to an embodiment of the present application is shown;
[0043] Figure 3 A third flow chart of the data deduplication method according to an embodiment of the present application is shown;
[0044] Figure 4 A fourth flow chart of the data deduplication method according to an embodiment of the present application is shown;
[0045] Figure 5 A structural schematic diagram of a data deduplication device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.
[0047] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0048] Hereinafter, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.
[0049] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.
[0050] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0051] The data deduplication method is described below with reference to some specific embodiments.
[0052] Figure 1 A flow chart of a data deduplication method according to an embodiment of the present application is shown. Exemplarily, the data deduplication method includes the following steps:
[0053] Step S100, obtaining a target data table corresponding to a deduplication task, each target data table being configured with at least one compression bitmap.
[0054] In this step, firstly, the upstream database data is read through Flink CDC, where the upstream database may include but is not limited to the business system Mysql database, log database, Oracle database, file database, etc. The data read through the upstream database is the target data table. After obtaining the target data table that needs to be deduplicated, at least one compression bitmap is configured for each target data table.
[0055] The compressed bitmap includes a plurality of bits having a preset number of bits, and each bit is provided with a corresponding state identifier.
[0056] The preset number of bits is the same as the preset number of bits for encoding the spliced characters below, which can be 32 bits, 64 bits or 8 bits, etc. The preset bitmap can be determined according to the needs of the user.
[0057] Step S200, determining the comparison field data of each target data table in each deduplication task.
[0058] In some implementations, after acquiring the data in these upstream databases, when configuring the task, first configure the comparison fields that need to be used for deduplication comparison in each target data table in the acquired data, where the comparison fields are the column names of each table, such as the name, card opening time, ID number, etc. in the bank card opening record table can all be used as comparison fields. After acquiring the comparison fields of each target data table, these comparison fields need to be saved, and they need to be saved with an identification mark to distinguish which target data table the comparison field belongs to.
[0059] In this step, after the deduplication task is started, the table names of the target data tables that need to be deduplicated are assigned to different deduplication tasks. It should be noted that one deduplication task corresponds to one thread, and one deduplication task can perform deduplication on multiple target data tables at the same time. When performing a deduplication task, the comparison field data of each target data table can be determined according to the comparison field of each target data table configured in the configuration task.
[0060] In some embodiments, Figure 2 As shown, step S200 includes:
[0061] Step S210 , according to the identification mark of each target data table in each deduplication task, the comparison field of each target data table is obtained from all the comparison fields with the identification mark saved.
[0062] Step S220 , extracting comparison field data from the corresponding target data table according to the comparison field of each target data table.
[0063] Specifically, in order to distinguish each target data table, each target data table is provided with a unique identification mark, and the identification mark of the target data table is the same as the identification mark of the comparison field set for each target data table. In this way, when deduplication processing is performed on the target data table, the corresponding comparison field can be known according to the table name in the deduplication task, and then the comparison field data can be extracted from the corresponding target data table according to the comparison field of the target data table.
[0064] For example, in one of the deduplication tasks, three target data tables need to be deduplicated. The three tables are Table 1, Table 2 and Table 3, as shown in the following table. Table 1 includes columns a1, b1, c1 and d1, Table 2 includes columns a2, b2, c2, d2 and e2, and Table 3 includes columns a3, b3, c3, d3 and e3.
[0065] Table 1 is:
[0066]
[0067] Table 2 is:
[0068]
[0069] Table 3 is:
[0070]
[0071] If, during task configuration, the comparison fields of Table 1 are configured as a1 and c1, the comparison fields of Table 2 are configured as b2, c2, and d2, and the comparison fields of Table 3 are configured as a3, b3, and c3, then during deduplication processing, for deduplication processing of Table 1, the comparison field data to be extracted based on the obtained comparison fields corresponding to Table 1 include aa1 and cc1 in the first row, aaaa1 and bbbb1 in the second row, aaa1 and ccc1 in the third row, etc.; for deduplication processing of Table 2, based on the obtained comparison fields corresponding to Table 1, For the comparison fields of Table 2, the comparison field data to be extracted include bb2, cc2 and dd2 in the first row, bbb2, ccc2 and ddd2 in the second row, bbbb2, cccc2 and dddd2 in the third row, etc.; for the deduplication processing of Table 3, according to the comparison fields of Table 3 obtained, the comparison field data to be extracted include aa3, bb3 and cc3 in the first row, aaa3, bbb3 and ccc3 in the second row, and aaaa3, bbbb3 and cccc3 in the fourth row.
[0072] Step S300, concatenate all comparison field data in the same record unit of each target data table to form a concatenated character, and encode the concatenated character according to a preset number of bits to obtain the comparison value corresponding to each record unit respectively.
[0073] Among them, the comparison value of the preset number of bits is a 32-bit comparison value or a 64-bit comparison value. In addition, the comparison value of the preset number of bits can also be 16 bits or 8 bits. However, due to the small storage capacity of 16 bits and 8 bits, the comparison value in the present application is preferably encoded as 32 bits or 64 bits. The specific preset number of bits is set to 32 bits or 64 bits can be determined as needed. If the amount of data is large, the preset position is set to 64 bits. If the amount of data is not too large and 32 bits are sufficient, it is preferably set to 32 bits.
[0074] The following description is made by taking the preset number of bits as 32 as an example.
[0075] Wherein, the same record unit is usually the same row of data in the excel table. After extracting the required comparison field data in step S200, the comparison field data of each row in the target data table are spliced in this step. For example, the comparison fields of Table 1 are a1 and c1, and the splicing characters after the comparison field data are spliced are aa1cc1, aaaa1cccc1, aaa1ccc1, etc. The comparison fields of Table 2 are b2, c2 and d2, and the splicing characters formed after the comparison field data are spliced are bb2cc2dd2, bbb2ccc2ddd2, bbbb2cccc2dddd2, etc., and then these splicing characters are CRC32 encoded respectively, and each splicing character is converted into a 32-bit comparison value, which is not only convenient for storing the comparison value in the efficient compression bitmap, but also convenient for finding whether the value exists in the compression bitmap later. It can be understood that the processing method of Table 3 is the same as that of Table 1 and Table 2, which will not be repeated here.
[0076] Step S400: Detect whether there is a bit matching the comparison value in each compressed bitmap.
[0077] In this step, when deduplication is performed on the target data table, it is usually performed in units of one record unit. After the comparison field data corresponding to the comparison field in the same record unit are concatenated and encoded to form a comparison value of a preset number of bits, it is detected whether there are bits in the compressed bitmap that match the comparison value.
[0078] Step S500: When a corresponding bit is matched for the first time, the state flag of the corresponding bit in the matched current compression bitmap is changed from a first preset value to a second preset value.
[0079] Step S600, when the corresponding bit is matched again, if the state identifier of the corresponding bit in any of the compressed bitmaps is the second preset value, it is determined that the record unit corresponding to the current comparison value is repeated data, and the repeated data is output.
[0080] In some implementations, each bit of the compressed bitmap represents a coded value of a preset number of bits; each bit is provided with a state identifier; and the state identifier includes a second preset value and a first preset value.
[0081] In order to facilitate the computer to distinguish and judge, the first preset value may be 0 and the second preset value may be 1.
[0082] Among them, each target data table corresponds to at least one compression bitmap; when the target data table is deduplicated, when the comparison value corresponding to a record unit in the target data table matches the corresponding bit a for the first time, the state flag of the bit is changed from the first preset value to the second preset value, and the comparison value is written into the bit a of the compression bitmap currently being written; when the comparison value corresponding to another record unit also matches the bit a again, at this time, since the state flag of the bit a of one of the compression bitmaps is the second preset value, it means that the data of the record unit is repeated with the previous data, and the repeated data is output to downstream processing.
[0083] It can be understood that each target data table corresponds to at least one compressed bitmap. In this embodiment, the preset number of bits is 32 as an example, and the bits of the compressed bitmap are also 32 bits. The compressed bitmap includes bits, each bit corresponds to a 32-bit encoding value. After the corresponding comparison field data of the corresponding target data table is spliced and encoded into a 32-bit comparison value, it is checked whether the bit corresponding to the comparison value in at least one compressed bitmap corresponding to the target data table is marked as the second preset value. If it is marked as the second preset value, it means that the data is repeated. When there is repeated data, the entire record of the repeated data (i.e. the corresponding row data) is pushed to the external message queue, and other data streams are used to process the data, such as issuing an alarm or ignoring it.
[0084] If the mark of the bit in the compressed bitmap corresponding to the comparison value is the first preset value, it means that there is no duplicate data.
[0085] In some implementations, a target data table includes at least two compressed bitmaps. After step S400, Figure 3 The process shown includes: step S700, when the corresponding bit positions are matched, if the state flags of the corresponding bit positions are all first preset values, it is determined whether the preset scrolling time of the currently written compressed bitmap has been reached.
[0086] If it is reached, step S800 is executed to write the current comparison value into the corresponding bit in the next compression bitmap corresponding to the current target data table, and modify the mark of the current bit from the first preset value to the second preset value.
[0087] Specifically, if the comparison value is searched in the corresponding compressed bitmap, if the bits corresponding to the comparison value in the corresponding multiple compressed bitmaps are all the first preset values, it means that the corresponding comparison value has not been written into the compressed bitmap, that is, there is no duplication. Since a target data table in the present application corresponds to multiple compressed bitmaps, when writing data to the compressed bitmap, it will be written in the order of the multiple compressed bitmaps. Therefore, if the corresponding bit is matched, it is determined whether the preset scrolling time of the compressed bitmap currently being written has been reached. If not, step S900 is executed to write the comparison value into the corresponding bit of the current compressed bitmap; if the compressed bitmap currently being written has reached the preset scrolling time, the corresponding comparison value is written into the corresponding bit of the next compressed bitmap. It should be noted that when data is written into the bit of the compressed bitmap, its mark is changed from the first preset value to the second preset value. For example, when we write data S for the first time, the encoded comparison value is 00000011. We find the bit corresponding to the comparison value. Since the data of S has not been written before, the status flag value of this bit is 0 (the first preset value) at this time. Then we change the status flag of this bit from 0 to 1, where 1 means that the data S has been written. When we write S for the second time, we find the bit of 00000011. Since S has been written once before, the status flag value of this bit has been set to 1 during the previous write, so it is considered to be duplicate data.
[0088] In some embodiments, in order to achieve deduplication of data at preset intervals, that is, to deduplicate data for one day, or 2 hours of data, or reverse deduplication, etc., the present application adopts a method in which one target data table corresponds to multiple compression bitmaps, and each compression bitmap sets a preset scrolling time according to a preset interval time. When the preset interval time is reached, the data in the first compression bitmap can be cleared according to the configuration, thereby achieving the purpose of deduplication of data at the preset interval time.
[0089] Specifically, when the preset scrolling time of the last compression bitmap corresponding to the current target data table is reached, the data in the bit position of the first compression bitmap corresponding to the current target data table is cleared, and all the second preset values of the previous first compression bitmap are modified to the first preset values; the current first compression bitmap is used as the last compression bitmap, and the second compression bitmap corresponding to the current target data table is used as the first compression bitmap, the remaining compression bitmaps are sorted in sequence, and the data of the current target data table is continued to be deduplicated until all the data in the current target data table is deduplicated.
[0090] It can be understood that the preset scrolling time is set by the user, which can be set during task configuration. Since the compressed bitmaps corresponding to the target data table are sorted in sequence, the data is also written in sequence. When the last compressed bitmap is written and the preset scrolling time of the last compressed bitmap is reached, the multiple compressed bitmaps corresponding to the target data table have reached the preset scrolling time. Then, the data of the first compressed bitmap corresponding to the target data table is cleared to facilitate the writing of the new comparison value, and the data at the preset time interval can be deduplicated. When clearing the data in the compressed bitmap, the status flags in the compressed bitmap are modified to the first preset value at the same time, and then the compressed bitmap is used as the last compressed bitmap. For example, the target data table 1 corresponds to compressed bitmap A, compressed bitmap B and compressed bitmap C, and the preset scrolling time is 1 hour. If there is no repeated data in the compressed bitmap, the comparison value is written into the compressed bitmap. In the 0-1 hour time period, it is first written into the compressed bitmap A. If it is in the 1-2 hour time period, when writing the comparison value, it is written into the compressed bitmap B. If it is in the 2-3 hour time period, when writing the comparison value, it is written into the compressed bitmap C. If the preset scrolling time of the compressed bitmap C ends, the data in the compressed bitmap A is cleared, and the status flag of the compressed bitmap A is modified to the first preset value, that is, the compressed bitmap A is used as the last compressed bitmap. When writing the comparison value to the compressed bitmap, it can be written into the compressed bitmap A. Then the compressed bitmap B can be used as the first compressed bitmap, and the compressed bitmap C can be used as the second compressed bitmap. If the target data table corresponds to two compressed bitmaps, including compressed bitmap A and compressed bitmap B, and the preset scrolling time of each compressed bitmap is 1 hour, then if the scrolling time is reached, the scrolling time is the sum of the preset scrolling times of the two compressed bitmaps. When the new comparison value is written, the data of the previous compressed bitmap will be cleared. That is, if the compressed bitmap being written is compressed bitmap B, but when the new comparison value needs to be written, the preset scrolling time of compressed bitmap B has been reached, then another compressed bitmap needs to be switched for writing. Therefore, the data of compressed bitmap A is cleared and switched to writing to compressed bitmap A. This can form a rotation storage and support the storage of at least 1 hour of data.
[0091] The following uses an example where a target data table corresponds to two compressed bitmaps and the preset rolling time of each compressed bitmap is 1 hour to illustrate the data deduplication method of the present application. The present application is mainly used for deduplication of real-time data stream intervals. Figure 4 As shown, after reading data from the upstream database 10 (business system Mysql database, log database, Oracle database, file database, etc.), enter the data reading and encoding process 20, which includes:
[0092] Step S21, distribute different table names to different deduplication tasks through the KeyBy operator.
[0093] In this step, after the deduplication task is started and data is obtained from the upstream database, different table names are distributed to different deduplication tasks through the KeyBy operator. In this way, a deduplication task can handle the deduplication tasks of several target data tables.
[0094] When performing deduplication processing, each deduplication task executes step S21 to obtain the comparison field corresponding to the target data table from the configuration parameters.
[0095] Specifically, when configuring the task, the comparison fields of each target data table have been configured. When the deduplication task performs deduplication processing on the data, the comparison fields of the corresponding target data table can be obtained from the configuration parameters.
[0096] Step S22, determining the comparison field data corresponding to the target data table according to the comparison field.
[0097] This step has been described in step S100 and will not be repeated here.
[0098] Step S23, after splicing all the comparison field data in the same row of each target data table, a splicing character is formed, and the splicing character is CRC32 encoded to be converted into a 32-bit comparison value.
[0099] This step is consistent with the above step S300 and will not be repeated here.
[0100] After obtaining the 32-bit comparison value, the repeated data processing flow 30 is entered, which specifically includes:
[0101] Step S31, determining whether the state flag of any bit corresponding to the comparison value in the two compressed bitmaps corresponding to the target data table is a second preset value.
[0102] This step is consistent with step S400 and will not be described again here.
[0103] If the bit corresponding to the comparison value in the compressed bitmap is already marked as the second preset value, it means that the comparison value has been written into the compressed bitmap, and the current comparison value is repeated with the previous value, and step S32 is executed to push the repeated data to the message queue for alarm or ignore processing. It can be understood that the repeated data is pushed to the message queue, not only for alarm or ignore processing, but also for other processing methods, which is not limited here, and the specific processing method can be set according to user needs.
[0104] If it is determined in step S31 that the status flags of the bits corresponding to the comparison value in the two compression bitmaps corresponding to the target data table are both the first preset values, it means that there is no duplicate data, and the non-duplicate data writing process 40 is entered, which specifically includes:
[0105] Step S41, determining whether the preset scrolling time of the currently written compressed bitmap has been reached.
[0106] In this example, one target data table corresponds to two compressed bitmaps. If the conclusion drawn in step S31 is that there is no duplicate data, the comparison value needs to be written into the compressed bitmap. When writing into the compressed bitmap, it is necessary to determine whether the preset scrolling time of the compressed bitmap currently being written has been reached. For example, if the preset scrolling time is 1 hour, and the compressed bitmap currently being written is compressed bitmap A, when judging, it is determined whether the compressed bitmap A has reached 1 hour from the time the first comparison value was written to the current time.
[0107] If it is determined in step S41 that the currently written compressed bitmap has reached the preset scrolling time, step S42 is executed to determine whether the data in another compressed bitmap needs to be cleared.
[0108] In this step, if it is determined in step S41 that the currently written compressed bitmap has reached the preset scrolling time, it is necessary to write to another compressed bitmap, and determine whether data has been written in the other compressed bitmap. If data has been written, it needs to be cleared first, so as to achieve the purpose of rotating written data.
[0109] If clearing is required, step S43 is executed to clear the data in another compressed bitmap, and the marks in another compressed bitmap are all changed to the first preset value, and then the comparison value is written into the corresponding bit of the compressed bitmap.
[0110] If clearing is not required, step S44 is executed to directly write the comparison value into another compressed bitmap.
[0111] If it is determined in step S41 that the currently written compressed bitmap has not reached the preset scrolling time, step S45 is executed to continue writing the comparison value into the current compressed bitmap.
[0112] For example, if there are two compressed bitmaps A and B, and the compressed bitmap currently being written is compressed bitmap A, when a new comparison value needs to be written, it is determined whether compressed bitmap A has reached the preset scrolling time. If so, it is determined whether compressed bitmap B has been written with data. If data has been written, compressed bitmap B is cleared, and then the comparison value is written into the corresponding bit of compressed bitmap B; if there is no data written into compressed bitmap B, it is directly switched to compressed bitmap B for writing; then after compressed bitmap B reaches the preset scrolling time, the data of compressed bitmap A is cleared, and the data is stored in rotation in sequence. If compressed bitmap A has not reached the preset scrolling time, it continues to write into compressed bitmap A. In this way, the two compressed bitmaps are used to store the marked data in rotation, and the data of a time period (1 hour) is stored in one of the compressed bitmaps. When the preset scrolling time is reached, the data in the other compressed bitmap is deleted, and the subsequent data is written into the other compressed bitmap, forming a rotating storage to support the storage of data for at least 1 time period (preset scrolling time).
[0113] After the comparison value is written into the compressed bitmap, the corresponding data is pushed to the target database 50, wherein the target database may be but is not limited to a Mysql database, an Oracle database, a Doris database, etc.
[0114] This application uses a compressed bitmap to deduplicate data by splicing the comparison field data and encoding it according to preset rules. The encoded comparison value can then quickly determine whether there is duplicate data within a period of time.
[0115] In addition, compared with the deduplication judgment of the prior art, the present application can save the memory used for real-time synchronous deduplication. Specifically, in the deduplication judgment of the prior art, each row of data needs to store multiple columns of data. In the present application, multiple columns of original data need to be compared during deduplication. The compressed bitmap storage can convert the field data of each row to be compared into a value of a preset number of bits through preset rules, and then fill it into the compressed bitmap bit, occupying only 1 bit of storage in the compressed bitmap. In addition, because the compressed bitmap has its own compression algorithm, 10 million data occupies about 1-10MB, which can save a lot of deduplication memory compared with the original data comparison.
[0116] Furthermore, the present application integrates the deduplication calculation process into the data calculation process to avoid writing redundant data into the target database, and deduplicates data during the synchronization process. Compared with the prior art of deduplicating data by using the database index mechanism when writing data to the database, it can effectively reduce the pressure of writing and updating data in the target database.
[0117] In addition, this application does not need to access external storage such as redis and hbase, which reduces external dependency on systems, saves external storage consumption, reduces interactions with storage systems, reduces system complexity, and improves system stability.
[0118] Figure 5 A schematic diagram of the structure of a data deduplication device according to an embodiment of the present application is shown. Exemplarily, the data deduplication device includes:
[0119] The acquisition module 100 is used to acquire the target data table corresponding to the deduplication task, and each target data table is configured with at least one compression bitmap.
[0120] The compressed bitmap includes a plurality of bits having a preset number of bits, and each bit is provided with a corresponding state identifier.
[0121] The determination module 200 is used to determine the comparison field data of each target data table in each deduplication task.
[0122] The encoding module 300 is used to concatenate all comparison field data in the same record unit of each target data table to form a concatenated character, and encode the concatenated character according to a preset number of bits to obtain the comparison value corresponding to each record unit.
[0123] The detection module 400 is used to detect whether there are bits in the compressed bitmap that match the comparison value.
[0124] The changing module 500 is used to change the state flag of the corresponding bit from the first preset value to the second preset value when the corresponding bit is matched for the first time.
[0125] The output module 600 is used to determine that the record unit corresponding to the current comparison value is duplicate data and output the duplicate data when the corresponding bit position is matched again if the state identifier of the corresponding bit position in any compressed bitmap is the second preset value.
[0126] It can be understood that the device of this embodiment corresponds to the data deduplication method of the above embodiment, and the optional items in the above embodiment are also applicable to this embodiment, so they will not be repeated here.
[0127] The present application also provides a terminal device. Exemplarily, the terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program, thereby enabling the terminal device to execute the functions of each module in the above-mentioned data deduplication method or the above-mentioned data deduplication device.
[0128] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application.
[0129] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.
[0130] The present application also provides a readable storage medium for storing the computer program used in the above-mentioned terminal device.
[0131] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0132] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0133] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.
[0134] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A data deduplication method, It is characterized in that include: Obtain a target data table corresponding to the deduplication task, each of the target data tables being configured with at least one compressed bitmap; wherein the compressed bitmap comprises a plurality of bits having a preset number of bits, and each of the bits is provided with a corresponding state identifier; Determine the comparison field data of each target data table in each deduplication task; splicing all comparison field data in the same record unit of each target data table to form a splicing character, and encoding the splicing character according to the preset number of bits to obtain the comparison value corresponding to each record unit respectively; Detecting respectively whether there are bits in each of the compressed bitmaps that match the comparison value; When the corresponding bit position is matched for the first time, the state flag of the corresponding bit position in the matched current compression bitmap is changed from the first preset value to the second preset value; When the corresponding bit position is matched again, if the state identifier of the corresponding bit position in any one of the compressed bitmaps is the second preset value, it is determined that the record unit corresponding to the current comparison value is repeated data, and the repeated data is output; Wherein, if each of the target data tables is configured with at least two of the compressed bitmaps; After respectively detecting whether there are bits in each of the compressed bitmaps that match the comparison value, the method further includes: When the corresponding bit positions are matched, if the state flags of the corresponding bit positions are all first preset values, determining whether a preset scrolling time of the currently written compressed bitmap has been reached; If so, the current comparison value is written into the corresponding bit in the next compression bitmap corresponding to the current target data table, and the state flag of the current bit is changed from the first preset value to the second preset value.
2. The data deduplication method according to claim 1, It is characterized in that The method further comprises: When the preset scrolling time of the last compression bitmap corresponding to the current target data table is reached, the data in the bit of the first compression bitmap corresponding to the current target data table is cleared, and all the second preset values of the first compression bitmap are changed to the first preset values; The current first compressed bitmap is used as the last compressed bitmap, and the second compressed bitmap corresponding to the current target data table is used as the first compressed bitmap. The remaining compressed bitmaps are sorted in sequence, and the data in the current target data table is continued to be deduplicated until all the data in the current target data table is deduplicated.
3. The data deduplication method according to claim 1, It is characterized in that Before determining the comparison field data of each target data table in each deduplication task, the method further includes: The comparison fields configured in each target data table are obtained, and all the comparison fields in the same target data table are saved with identification marks.
4. The data deduplication method according to claim 3, It is characterized in that The determining of the comparison field data of each target data table in each deduplication task includes: According to the identification mark of each target data table in each deduplication task, obtaining the comparison field of each target data table from all the comparison fields with the identification mark saved; According to the comparison field of each target data table, comparison field data is extracted from the corresponding target data table.
5. The data deduplication method according to claim 1, It is characterized in that The preset number of bits of comparison value is a 32-bit comparison value or a 64-bit comparison value.
6. The data deduplication method according to claim 1, It is characterized in that The second preset value is 1; the first preset value is 0.
7. A data deduplication device, It is characterized in that include: An acquisition module, used for acquiring a target data table corresponding to a deduplication task, each of the target data tables being configured with at least one compressed bitmap; wherein the compressed bitmap comprises a plurality of bits having a preset number of bits, and each of the bits is provided with a corresponding state identifier; A determination module, used to determine the comparison field data of each target data table in each deduplication task; An encoding module, used for splicing all comparison field data in the same record unit of each target data table to form a splicing character, and encoding the splicing character according to the preset number of bits to obtain the comparison value corresponding to each record unit respectively; A detection module, used for detecting whether there is a bit matching the comparison value in each of the compressed bitmaps; A changing module, used for changing the state flag of the corresponding bit from a first preset value to a second preset value when the corresponding bit is matched for the first time; an output module, configured to determine that the record unit corresponding to the current comparison value is duplicate data and output the duplicate data when the corresponding bit position is matched again if the state identifier of the corresponding bit position in any one of the compressed bitmaps is the second preset value; Wherein, if each of the target data tables is configured with at least two of the compressed bitmaps; In the detection module, after respectively detecting whether there are bits matching the comparison value in each of the compressed bitmaps, it is also used to determine whether, when matching the corresponding bits, if the state identifiers of the corresponding bits are all first preset values, whether the preset scrolling time of the currently written compressed bitmap has been reached; if so, the current comparison value is written into the corresponding bit in the next compressed bitmap corresponding to the current target data table, and the state identifier of the current bit is changed from the first preset value to the second preset value.
8. A terminal device, It is characterized in that The terminal device includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the data deduplication method according to any one of claims 1 to 6.
9. A computer-readable storage medium, It is characterized in that It stores a computer program, which, when executed on a processor, implements the data deduplication method according to any one of claims 1-6.
Citation Information
Patent Citations
Data storage using bitmaps
CN102511031A
Disk storage method and apparatus applied to distributed file system, and distributed file system
CN108021562A
Data processing method and device and server
CN110489405A
Business processing method and device, electronic equipment and storage medium
CN113468226A
Data deduplication method and device, equipment and storage medium
CN115794788A
