A data deduplication method, apparatus, and computer-readable storage medium
By combining Redis's ZSet and Set structures, the target window is retrieved at a preset period and deduplicated key strings are removed, solving the problem of resource waste and business crashes caused by duplicate messages in streaming data processing, and achieving efficient data deduplication.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI WEIMOB ENTERPRISE DEV CO LTD
- Filing Date
- 2022-10-10
- Publication Date
- 2026-05-26
AI Technical Summary
Existing streaming data processing fails to aggregate and deduplicate multiple duplicate messages within a short period, leading to resource waste and crashes in downstream message processing services.
By using Redis's ZSet structure to retrieve the target window at a preset period, and leveraging the unique property of the Set structure, duplicate key strings are generated and deduplicated to achieve message aggregation and deduplication.
It improves resource utilization and avoids resource shortages and waste caused by redundant processing when dealing with large amounts of data.
Smart Images
Figure CN115563094B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data, and in particular to a data deduplication method, apparatus and computer-readable storage medium. Background Technology
[0002] Streaming data is a sequence of data that arrives sequentially, in large quantities, rapidly, and continuously. Generally, streaming data can be viewed as a dynamic collection of data that grows indefinitely over time, and it has applications in fields such as network monitoring, sensor networks, aerospace, meteorological monitoring and control, and financial services. Streaming data requires incremental processing either record-by-record or according to a sliding time window, and can be used for various analyses, including correlation, aggregation, filtering, and sampling. The information obtained from such analyses provides in-depth insights into the data, enabling rapid responses to new situations.
[0003] Existing streaming data processing does not aggregate and deduplicate multiple duplicate messages within a short period of time; when the data volume is large, it can easily cause downstream message processing services to crash, and repeating the processing of duplicate data will result in a waste of resources.
[0004] In view of the above problems, designing a data deduplication method to improve resource utilization is an urgent problem to be solved by technical personnel in this field. Summary of the Invention
[0005] The purpose of this application is to provide a data deduplication method, apparatus, and computer-readable storage medium.
[0006] To address the aforementioned technical problems, this application provides a data deduplication method, comprising:
[0007] The target window in the ZSet structure is obtained according to a preset period; wherein, the target window is the first window in the ZSet structure arranged according to preset requirements.
[0008] Determine whether the target window meets the preset conditions;
[0009] If not, read all the key strings in the Set structure in the target window to use the Set structure to deduplicate messages;
[0010] The key string is a plurality of key strings containing the message generated by the message queue.
[0011] Preferably, before acquiring the target window in the ZSet structure at a preset period, the method further includes:
[0012] The window is created in chronological order using the ZSet structure;
[0013] The project name of each window is the creation time of each window.
[0014] Preferably, determining whether the target window meets preset conditions includes:
[0015] Determine whether the sum of the creation time and duration of the target window is not less than the current physical time;
[0016] If not, the target window does not meet the preset conditions, and the process proceeds to the step of reading all the key strings in the Set structure of the target window.
[0017] Preferably, the specific steps for the message queue to generate the key string include:
[0018] Retrieve the message from the message queue;
[0019] The messages are filtered according to preset rules;
[0020] The filtered message generates the key string, which is then sent to the window whose creation time has the smallest difference from the current physical time.
[0021] Preferably, before reading all the key strings in the Set structure of the target window, the method further includes:
[0022] Obtain the window lock of the target window.
[0023] Preferably, after reading all the key strings in the Set structure of the target window, the method further includes:
[0024] Restore the key string to the original data of the message;
[0025] Send the raw data to the message queue.
[0026] Preferably, after sending the raw data to the message queue, the method further includes:
[0027] Delete the target window and create a new one;
[0028] Release the window lock of the target window;
[0029] Return to the step of obtaining the target window in the ZSet structure according to the preset period.
[0030] To address the aforementioned technical problems, this application also provides a data deduplication device, comprising:
[0031] The acquisition module is used to acquire target windows in the ZSet structure at preset periods; wherein, the target window is the first window in the ZSet structure sorted according to preset requirements.
[0032] The judgment module is used to determine whether the target window meets preset conditions; if not, the reading module is triggered.
[0033] The reading module is used to read all the key strings in the Set structure in the target window, so as to deduplicate messages through the Set structure;
[0034] The key string is a plurality of key strings containing the message generated by the message queue.
[0035] To solve the above-mentioned technical problems, this application also provides another data deduplication device, including:
[0036] Memory, used to store computer programs;
[0037] A processor is configured to implement the steps of the data deduplication method described above when executing the computer program.
[0038] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data deduplication method described above.
[0039] The data deduplication method provided in this application obtains the target window in a ZSet structure at preset intervals. The target window is the first window in the ZSet structure arranged according to preset requirements. It determines whether the target window meets preset conditions; if not, it reads all key strings from the Set structure within the target window for message deduplication using the Set structure. The key strings are multiple key strings containing messages generated by a message queue. Therefore, the above scheme generates key strings from messages in the message queue and utilizes the deduplication mechanism of the Set structure of the window to deduplicate key strings containing messages. This achieves aggregated deduplication of data, solving the problem of resource waste caused by data processing constraints and redundant data processing when dealing with large amounts of data, thus improving resource utilization.
[0040] In addition, embodiments of this application also provide a data deduplication device and a computer-readable storage medium, which have the same effect as above. Attached Figure Description
[0041] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 A flowchart illustrating a data deduplication method provided in this application embodiment;
[0043] Figure 2 A flowchart of another data deduplication method provided in the embodiments of this application;
[0044] Figure 3 A schematic diagram illustrating the correspondence between windows and their key strings provided in embodiments of this application;
[0045] Figure 4 This is a schematic diagram of the structure of a data deduplication device provided in an embodiment of this application;
[0046] Figure 5 This is a schematic diagram of another data deduplication device provided in an embodiment of this application. Detailed Implementation
[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0048] The core of this application is to provide a data deduplication method, apparatus, and computer-readable storage medium.
[0049] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0050] Streaming data is a sequence of data that arrives sequentially, in large quantities, rapidly, and continuously. Generally, streaming data can be viewed as a dynamic data set that grows infinitely over time, and it has applications in network monitoring, sensor networks, aerospace, meteorological monitoring and control, and financial services. Streaming data requires incremental processing, either record-by-record or according to a sliding time window, and can be used for various analyses, including correlation, aggregation, filtering, and sampling. Information derived from such analyses provides in-depth understanding of the data, enabling rapid responses to new situations. Current streaming data processing methods do not aggregate and deduplicate multiple duplicate messages within a short period; this can easily lead to the collapse of downstream message processing operations when the data volume is large, and repeatedly processing duplicate data results in wasted resources.
[0051] Therefore, this application provides a data deduplication method based on a remote dictionary server (Redis) for storage and using a message queue (MQ) for asynchronous message reception and transmission. Figure 1 A flowchart of a data deduplication method provided in an embodiment of this application is shown below. Figure 1 As shown, the method includes:
[0052] S10: Obtain the target window in the ZSet structure according to a preset period; where the target window is the first window in the ZSet structure arranged according to preset requirements.
[0053] S11: Determine whether the target window meets the preset conditions; if not, proceed to step S12.
[0054] S12: Read all key strings from the Set structure in the target window for message deduplication using the Set structure. The key strings are multiple key strings containing the message, generated by the message queue.
[0055] As is understandable, Redis is a key-value storage system that supports a wider range of value types, including strings, lists, sets, sorted sets (ZSets), and hashes. The ZSet structure adds a "score" field to the Set structure, using score and index for sorting. Each element has a score, and all elements are arranged according to their scores, essentially forming a sorted linked list. In this embodiment, the windows in the ZSet structure are an ordered set of windows arranged according to preset requirements. Therefore, to achieve data deduplication, the windows must first be retrieved in a specific order. Specifically, the first window in the ZSet structure arranged according to preset requirements (the target window) is retrieved at a preset period, and the message data in the target window is then deduplicated. In this embodiment, the preset period is not limited and depends on the specific implementation; the preset requirement of arranging all windows in the ZSet structure is also not limited and depends on the specific implementation.
[0056] Furthermore, after obtaining the target window in the ZSet structure, it is determined whether the target window meets preset conditions. In this embodiment, the preset conditions are not limited; they can be determined by whether the creation time of the target window meets the requirements, or by whether the message content in the target window meets the requirements, depending on the specific implementation. If the target window does not meet the preset conditions, it is determined that the message data in the target window can be deduplicated; that is, all key strings in the Set structure of the target window are read. The Set structure is part of Redis, an interface class inherited from the collector; the Set structure only stores key strings, and the key must be unique. Therefore, the keys in the Set cannot be duplicated. Thus, the main function of the Set is to deduplicate the elements in the set, thereby achieving deduplication of the key strings containing messages in the target window.
[0057] It is important to note that in this embodiment, the key strings are multiple key strings containing messages generated by a message queue. A message queue (MQ) is a basic data structure generally used to solve problems such as application decoupling, asynchronous messaging, and traffic shaping, achieving a high-performance, highly available, scalable, and eventually consistent architecture. This embodiment does not limit the specific steps for generating key strings using the message queue; it depends on the specific implementation.
[0058] In this embodiment, the target window in the ZSet structure is obtained at a preset period. The target window is the first window in the ZSet structure arranged according to preset requirements. It is determined whether the target window meets the preset conditions. If not, all key strings in the Set structure of the target window are read for message deduplication using the Set structure. The key strings are multiple key strings containing messages generated by the message queue. Therefore, the above scheme generates key strings for messages in the message queue and uses the deduplication mechanism of the non-repeating characteristic of the Set structure of the window to deduplicate the key strings containing messages. This achieves aggregated deduplication of data, solving the problem of resource waste caused by tight data processing resources and redundant data processing when dealing with large amounts of data, thus improving resource utilization.
[0059] Figure 2 A flowchart illustrating another data deduplication method provided in an embodiment of this application. For example... Figure 2 As shown, before retrieving the target window from the ZSet structure at preset intervals, the method further includes:
[0060] S13: Create windows in chronological order using a ZSet structure; where the item name of each window is the creation time of that window.
[0061] As described in the above embodiments, the window is stored using a Redis ZSet structure, which is an ordered linked list. The window is also used to store the key strings to be deduplicated generated by the message queue (MQ). Therefore, in this embodiment, to achieve message deduplication, a window needs to be created before retrieving the target window from the ZSet structure at a preset period.
[0062] Specifically, windows are created in chronological order using a ZSet structure. These windows are consecutive on the timeline but do not repeat. The item name for each window is its creation time. Simultaneously, the window's creation time is used as the item name for the window within the Redis Set structure to store the message's key string, thus leveraging the inherent non-repeating characteristic of the Set structure to deduplicate messages. Figure 3 This is a schematic diagram illustrating the correspondence between windows and their key strings provided in embodiments of this application. For example... Figure 3 As shown, the windows in the ZSet structure are sorted in ascending order of time. For example, two windows with project names 1629617800000 and 1629617810000 are consecutive in time. Each window contains multiple key strings generated by MQ. For example, the Set structure in the window with project name 1629617850000 stores four key strings: key_1, key_2, key_3, and key_4.
[0063] In this embodiment, windows are created in chronological order using a ZSet structure, and the item name of each window is the creation time of that window, thus realizing the creation of windows and the storage of key strings.
[0064] Based on the above embodiments:
[0065] As a preferred embodiment, determining whether the target window meets preset conditions includes:
[0066] Determine if the sum of the creation time and duration of the target window is not less than the current physical time;
[0067] If not, the target window does not meet the preset conditions, and the process proceeds to the step of reading all key strings in the Set structure of the target window.
[0068] In the above embodiments, the preset conditions are not limited and are determined according to the specific implementation. As a preferred embodiment, in this embodiment, determining whether a target window meets the preset conditions requires judging whether the sum of the target window's creation time and duration is not less than the current physical time. It is understood that each window has a creation time, and since the windows are continuous on the timeline, each window also has a duration. Specifically, the sum of the target window's creation time and duration is compared with the current physical time. If the sum of the target window's creation time and duration is not less than the current physical time, it means the target window has not expired; if the sum of the target window's creation time and duration is less than the current physical time, it means the target window has expired, and the content in the target window needs to be processed, i.e., deduplicating the key strings in the Set structure.
[0069] In this embodiment, by determining that the sum of the creation time and duration of the target window is less than the current physical time, it is determined that the target window does not meet the preset conditions, and thus proceeds to the step of reading all the key strings in the Set structure of the target window to achieve data deduplication.
[0070] Based on the above embodiments:
[0071] As a preferred embodiment, the specific steps for generating key strings in a message queue include:
[0072] Retrieve messages from the message queue;
[0073] Filter messages according to preset rules;
[0074] The filtered messages generate a key string, which is then used to send the key string to the window with the smallest difference between the creation time and the current physical time.
[0075] In the above embodiments, the specific steps for generating key strings in the message queue are not limited and depend on the specific implementation. As a preferred embodiment, in this embodiment, messages are first consumed from the MQ, i.e., messages are read from the MQ, and the messages are initially filtered according to preset rules. In this embodiment, the preset rules are not limited, and the specific implementation method is adjusted according to different business needs. The purpose of this initial filtering is to filter out only the message content of interest, for example, only messages containing "A", while filtering out other messages, such as those containing "B" or "C".
[0076] Furthermore, the message content of interest is used to generate key strings for deduplication. In this embodiment, there are no restrictions on the specific generation method of the key strings; a custom generation method can be used, as long as the generated key strings can be parsed back into the original message. After generating the key strings, they are sent to the window with the smallest difference between the creation time and the current physical time, i.e., the window whose creation time is closest to the current physical time. This key string serves as the basis for deduplication and is unique within a single window. If two messages generate the same key string and are in the same window, the deduplication purpose can be achieved, and only one record will be stored in the window.
[0077] In this embodiment, by acquiring messages from the message queue, filtering the messages according to preset rules, and sending the key strings generated from the filtered messages to the window with the smallest difference between the creation time and the current physical time, the generation and storage of key strings in the window are realized.
[0078] like Figure 2 As shown, to control resource access, only one program can process the locked resource at a time. Before reading all the key strings in the Set structure of the target window, the following steps are also included:
[0079] S14: Acquire the window lock of the target window.
[0080] In this embodiment, by acquiring the window lock of the target window before reading all the key strings in the Set structure of the target window, the target window is locked, thereby achieving resource access control over the target window and making it only used for deduplication of key strings.
[0081] like Figure 2 As shown, in order to return the deduplicated message data, after reading all the key strings in the Set structure of the target window, the following steps are also included:
[0082] S15: Restore the key string to the original data of the message.
[0083] S16: Send raw data to the message queue.
[0084] Understandably, all key strings in the Set structure within the target window are read, and the inherent deduplication property of the Set structure is utilized to deduplicate the key strings in the target window. In this implementation, the deduplicated key strings are restored to the original message data, and the original data is sent to the MQ (Message Queuing). The MQ then regenerates the key strings from the messages and sends them to the latest window for subsequent message deduplication.
[0085] like Figure 2 As shown, in order to terminate the process of the target window that has completed deduplication and save access resources, after sending the original data to the message queue, the following steps are also included:
[0086] S17: Delete the target window and create a new window.
[0087] S18: Release the window lock of the target window and return to step S10.
[0088] It is understandable that when the target window completes message deduplication, restores the key strings to the original message data, and sends it to the message queue, the current target window has completed the message deduplication task. Therefore, in this embodiment, the deduplicated target window is deleted, and a new window is created in the same way as the window creation method in the above embodiment; the window lock of the target window is released, and resource access control on the current window is removed. Finally, the process returns to step S10: retrieve the target window in the ZSet structure according to a preset period, that is, perform message deduplication for the next target window.
[0089] In this embodiment, after sending the original data to the message queue, the target window is deleted and a new window is created. The window lock of the target window is released, and the process returns to the step of retrieving the target window from the ZSet structure at a preset period. This completes the process of deduplicating the target window, saves access resources, and starts message deduplication for the next target window.
[0090] In the above embodiments, the data deduplication method has been described in detail. This application also provides embodiments corresponding to the data deduplication device. It should be noted that this application describes the embodiments of the device from two perspectives: one is based on the functional modules, and the other is based on the hardware structure.
[0091] Figure 4 This is a schematic diagram of a data deduplication device provided in an embodiment of this application. Figure 4 As shown, the data deduplication device includes:
[0092] The acquisition module 10 is used to acquire the target window in the ZSet structure at a preset period; wherein, the target window is the first window in the ZSet structure that is sorted according to preset requirements.
[0093] The judgment module 11 is used to determine whether the target window meets the preset conditions; if not, the reading module is triggered.
[0094] The reading module 12 is used to read all the key strings in the Set structure in the target window, so as to deduplicate messages through the Set structure.
[0095] Among them, the key strings are multiple key strings containing messages generated through the message queue.
[0096] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.
[0097] Figure 5 This is a schematic diagram of another data deduplication device provided in an embodiment of this application, as shown below. Figure 5 As shown, the data deduplication device includes:
[0098] Memory 20 is used to store computer programs.
[0099] The processor 21 is configured to implement the steps of the data deduplication method mentioned in the above embodiments when executing a computer program.
[0100] The data deduplication device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.
[0101] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.
[0102] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the data deduplication method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the data deduplication method.
[0103] In some embodiments, the data deduplication device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.
[0104] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the data deduplication device and may include more or fewer components than shown.
[0105] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.
[0106] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0107] The foregoing has provided a detailed description of a data deduplication method, apparatus, and computer-readable storage medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.
[0108] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A data deduplication method, characterized in that, include: The target window in the ZSet structure is obtained according to a preset period; wherein, the target window is the first window in the ZSet structure arranged according to preset requirements. Determine whether the target window meets the preset conditions; If not, read all the key strings in the Set structure in the target window to use the Set structure to deduplicate messages; Wherein, the key string is a plurality of key strings containing the message generated by the message queue; Before acquiring the target window in the ZSet structure at a preset period, the method further includes: The window is created in chronological order using the ZSet structure; The project name of each window is the creation time of each window; The step of determining whether the target window meets the preset conditions includes: Determine whether the sum of the creation time and duration of the target window is not less than the current physical time; If not, the target window does not meet the preset conditions, and the process proceeds to the step of reading all the key strings in the Set structure of the target window.
2. The data deduplication method according to claim 1, characterized in that, The specific steps for the message queue to generate the key string include: Retrieve the message from the message queue; The messages are filtered according to preset rules; The filtered message generates the key string, which is then sent to the window whose creation time has the smallest difference from the current physical time.
3. The data deduplication method according to claim 1, characterized in that, Before reading all the key strings in the Set structure in the target window, the following is also included: Obtain the window lock of the target window.
4. The data deduplication method according to claim 3, characterized in that, After reading all the key strings in the Set structure of the target window, the process also includes: Restore the key string to the original data of the message; Send the raw data to the message queue.
5. The data deduplication method according to claim 4, characterized in that, After sending the raw data to the message queue, the method further includes: Delete the target window and create a new one; Release the window lock of the target window; Return to the step of obtaining the target window in the ZSet structure according to the preset period.
6. A data deduplication device, characterized in that, include: The acquisition module is used to acquire target windows in the ZSet structure at preset periods; wherein, the target window is the first window in the ZSet structure sorted according to preset requirements. The judgment module is used to determine whether the target window meets preset conditions; if not, the reading module is triggered. The reading module is used to read all the key strings in the Set structure in the target window, so as to deduplicate messages through the Set structure; Wherein, the key string is a plurality of key strings containing the message generated by the message queue; Before acquiring the target window in the ZSet structure at a preset period, the method further includes: The window is created in chronological order using the ZSet structure; The project name of each window is the creation time of each window; The step of determining whether the target window meets the preset conditions includes: Determine whether the sum of the creation time and duration of the target window is not less than the current physical time; If not, the target window does not meet the preset conditions, and the process proceeds to the step of reading all the key strings in the Set structure of the target window.
7. A data deduplication device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data deduplication method as described in any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data deduplication method as described in any one of claims 1 to 5.