Real-time data deduplication method and device based on Flink, electronic equipment and medium

Abstract

The invention discloses a real-time data deduplication method and device based on Flink, electronic equipment and a medium, and the method comprises the steps: determining a deduplication rule and a control strategy, and submitting an Flink task; acquiring real-time data in the message queue as an input data stream, determining a key value of the input data stream according to a deduplication rule, and further grouping the input data stream according to the key value through a grouping operator to obtain a grouped data stream; performing data deduplication on the grouped data stream according to the key value through a deduplication operator to obtain a main data stream and a side stream; and outputting the main data stream, and judging whether to store the side stream or not according to the control strategy. Real-time data deduplication is achieved based on the Flink, the data deduplication efficiency and reliability are improved, and the method can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a real-time data deduplication method, device, electronic device and medium based on Flink. Background Art

[0002] In the era of big data, with the acceleration of digital transformation, all walks of life have generated unprecedented amounts of data, covering a wide range from financial transactions to medical records, from social media interactions to industrial IoT sensor outputs. However, the explosive growth of data has also brought unprecedented challenges, especially the redundancy and duplication of data. Duplicate data not only wastes valuable storage space, but also increases the time and cost of data processing. More importantly, they reduce the accuracy and timeliness of data analysis, affecting the quality and efficiency of enterprise decision-making.

[0003] In response to the demand for real-time data deduplication, traditional solutions have their own advantages, but they all have obvious limitations in practical applications. Traditional deduplication methods such as feature engineering, combined with external storage (such as Redis), Bloom filter, bitmap-based deduplication, hash table-based, etc., although they can solve the problem of data duplication to a certain extent, often expose their own significant shortcomings in practical applications. For example: feature engineering is relatively complex in the implementation process, and often requires the writing of complex logic to handle duplicate items in the data, which increases the overall complexity and the possibility of errors. In addition, feature engineering depends on specific data sets and may introduce new errors, accompanied by performance issues. The method of combining external storage (such as Redis) will introduce additional overhead in network communication. This method not only increases the external dependence and complexity of the system, but also makes the solution inflexible and increases the cost of use and maintenance. The method based on bitmap deduplication has certain shortcomings in terms of memory consumption and scalability, data type support, flexibility and customization, performance and latency, and dependency and complexity. Deduplication based on hash tables cannot handle large data volumes and real-time scenarios.

[0004] The above problems need to be solved urgently.

[0005] Terminology explanation:

[0006] Flink: Apache Flink is a distributed stream processing and batch processing framework that aims to provide an efficient, scalable, and fault-tolerant processing engine that supports large-scale data processing and real-time data analysis.

[0007] Linked list: A common linear data structure, which consists of a series of nodes (Node). Each node contains a data part and a reference (or pointer) to the next node. The nodes in the linked list are connected to each other through pointers to form a chain structure.

[0008] Bit array: A compact data structure where each element occupies only one bit and can represent two states (0 and 1). Therefore, bit arrays have very high storage efficiency.

[0009] Bitmap: A bitmap is a data structure used to store data in the form of bits. Each bit represents whether a specific element exists. It is commonly used in data deduplication scenarios.

[0010] Bit: The most basic unit in computer science, the smallest unit used to represent information.

[0011] Hash: An algorithm that maps data of arbitrary length to a fixed-length output. A hash function takes input data (often called a "message" or "key") and generates a fixed-size string called a hash value or digest.

[0012] Time window: A concept in the field of data processing and analysis, especially in stream processing systems (such as Apache Flink), used to segment continuous data streams. Time windows divide infinite data streams into "windows" of finite size, allowing the system to aggregate, calculate, and analyze within these windows.

[0013] Sliding step size: In data processing or algorithms, the distance of each movement or sliding; in this patent, it specifically refers to the distance of sliding within the time window. Summary of the invention

[0014] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0015] To this end, an object of an embodiment of the present invention is to provide a real-time data deduplication method based on Flink, which implements real-time data deduplication based on Flink and improves the efficiency and reliability of data deduplication.

[0016] Another object of an embodiment of the present invention is to provide a real-time data deduplication device based on Flink.

[0017] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0018] On the one hand, an embodiment of the present invention provides a real-time data deduplication method based on Flink, comprising the following steps:

[0019] Determine deduplication rules and control strategies, and submit Flink tasks;

[0020] Acquire real-time data in the message queue as an input data stream, determine a key value of the input data stream according to the deduplication rule, and then group the input data stream according to the key value through a grouping operator to obtain a grouped data stream;

[0021] Deduplication is performed on the packet data stream according to the key value by using a deduplication operator to obtain a main data stream and a side stream;

[0022] The main data stream is output, and whether to save the side stream is determined according to the control strategy.

[0023] Furthermore, in one embodiment of the present invention, the deduplication rule includes several data types and corresponding deduplication fields, and the control strategy includes control parameters and storage strategy.

[0024] Further, in one embodiment of the present invention, determining the key value of the input data stream according to the deduplication rule specifically includes:

[0025] Determine a current data type of the input data stream, and determine a current deduplication field corresponding to the current data type according to the deduplication rule;

[0026] Reading a deduplication field value of the input data stream according to the current deduplication field, and splicing the deduplication field value to obtain a spliced ​​field value;

[0027] The key value is determined according to the hash value of the concatenated field value.

[0028] Further, in one embodiment of the present invention, the step of grouping the input data stream according to the key value by a grouping operator to obtain a grouped data stream specifically includes:

[0029] Implement Flink's KeySelector interface based on the grouping operator;

[0030] Several input data streams with the same key value are divided into a group to obtain the grouped data streams corresponding to the key value.

[0031] Further, in one embodiment of the present invention, the deduplication operator is used to perform data deduplication on the packet data stream according to the key value to obtain the main data stream and the side stream, which specifically includes:

[0032] Inherit Flink's KeyedProcessFunction abstract class through the deduplication operator;

[0033] Determine the time window and the sliding step according to the control parameters of the control strategy, create a linked list, and then encapsulate the bit array and the time record through the nodes of the linked list;

[0034] Reading the packet data stream through the time window, and determining whether a current key value corresponding to a current data stream in a current time window exists in the linked list;

[0035] If the current key value does not exist in the linked list, collect the current data stream into the main data stream and add the current key value to the node of the linked list; if the current key value exists in the linked list, collect the current data stream into the side stream;

[0036] The current time window is updated according to the sliding step, and the step of returning to determine whether the current key value corresponding to the current data stream of the current time window exists in the linked list is returned.

[0037] Further, in one embodiment of the present invention, the adding the current key value to the node of the linked list specifically includes:

[0038] When the linked list is empty, the current key value is mapped to the bit array of the first node of the linked list, and the time record of the first node is updated according to the time of the current data stream and the sliding step;

[0039] When the linked list is not empty, determining whether the time of the current data stream is less than the time record of the tail node of the linked list;

[0040] If the time of the current data flow is less than or equal to the time record of the tail node of the linked list, the current key value is mapped to the bit array of the tail node of the linked list; if the time of the current data flow is greater than the time record of the tail node of the linked list, the current key value is mapped to a newly added node of the linked list, and the time record of the newly added node is determined based on the time record of the tail node and the sliding step.

[0041] Further, in one embodiment of the present invention, the determining whether the current key value corresponding to the current data stream of the current time window exists in the linked list specifically includes:

[0042] Traversing all nodes of the linked list, calculating the mapping subscript of the current key value at each position in the bit array of each node;

[0043] When the values ​​of the mapping subscripts of the current key value at each position in the bit array of a certain node are all 1, it is determined that the current key value exists in the linked list.

[0044] On the other hand, an embodiment of the present invention provides a real-time data deduplication device based on Flink, including:

[0045] The task submission module is used to determine deduplication rules and control strategies and submit Flink tasks;

[0046] A data grouping module, used to obtain real-time data in the message queue as an input data stream, and determine a key value of the input data stream according to the deduplication rule, and then group the input data stream according to the key value through a grouping operator to obtain a grouped data stream;

[0047] A data deduplication module, configured to perform data deduplication on the packet data stream according to the key value by using a deduplication operator to obtain a main data stream and a side stream;

[0048] A data output module is used to output the main data stream and determine whether to save the side stream according to the control strategy.

[0049] On the other hand, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the program is executed by the processor, the real-time data deduplication method based on Flink as described above is realized.

[0050] On the other hand, an embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the real-time data deduplication method based on Flink as described above.

[0051] The advantages and beneficial effects of the present invention will be partly given in the following description, partly become apparent from the following description, or be understood through the practice of the present invention:

[0052] The embodiment of the present invention determines deduplication rules and control strategies, and submits Flink tasks, obtains real-time data in a message queue as an input data stream, and determines the key value of the input data stream according to the deduplication rules, and then groups the input data stream according to the key value through a grouping operator to obtain a grouped data stream, performs data deduplication on the grouped data stream according to the key value through a deduplication operator to obtain a main data stream and a side stream, outputs the main data stream, and determines whether to save the side stream according to the control strategy. The embodiment of the present invention determines the key value of the input data stream according to the deduplication rules, groups the input data stream according to the key value through a grouping operator, performs data deduplication on the grouped data stream according to the key value through a deduplication operator, realizes real-time data deduplication based on Flink, and improves the efficiency and reliability of data deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A step flow chart of a real-time data deduplication method based on Flink provided in an embodiment of the present invention;

[0055] Figure 2 A specific flow chart of the real-time data deduplication method based on Flink provided in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a deduplication rule provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a control strategy provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of a message queue provided by an embodiment of the present invention;

[0059] Figure 6 A step flow chart of step S102 provided in an embodiment of the present invention;

[0060] Figure 7 Another step flow chart of step S102 provided in an embodiment of the present invention;

[0061] Figure 8 A schematic diagram before and after data grouping provided by an embodiment of the present invention;

[0062] Fig. 9A step flow chart of step S103 provided in an embodiment of the present invention;

[0063] Fig.10 A schematic diagram of data before and after deduplication provided by an embodiment of the present invention;

[0064] Fig.11 A step flow chart of step S1034 provided in an embodiment of the present invention;

[0065] Fig.12 A schematic diagram of processing data streams based on time windows and linked lists provided in an embodiment of the present invention;

[0066] Fig.13 A schematic diagram of bit array mapping of linked list nodes in a time window provided by an embodiment of the present invention;

[0067] Fig.14 A step flow chart of step S1033 provided in an embodiment of the present invention;

[0068] Fig.15 A data interaction diagram of a real-time data deduplication method based on Flink provided in an embodiment of the present invention.

[0069] Fig.16 A schematic diagram of the structure of a real-time data deduplication device based on Flink provided in an embodiment of the present invention;

[0070] Fig.17 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention;

[0071] Fig.18 A schematic diagram of the structure of a storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limitations on the present application. It should be noted that, although the functional module division is performed in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order from the module division in the system schematic diagram or the flow chart. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0073] In the description of the present invention, the meaning of "a plurality" is two or more. If there is a description of the first or the second, it is only for the purpose of distinguishing the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this document are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0074] The real-time data deduplication method based on Flink provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the real-time data deduplication method based on Flink, etc., but is not limited to the above forms.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0076] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0077] In view of the problems existing in existing technologies, a new type of real-time data deduplication technology is urgently needed. It should not only overcome the limitations of existing methods, such as reducing complexity, reducing dependence on external storage, improving scalability and processing efficiency, but also be able to adapt to the ever-changing data environment and ensure the real-time and accuracy of data processing. This technology should have high performance, high concurrent processing capabilities, and be able to be seamlessly integrated into the existing data processing pipeline. At the same time, it should also be easy to maintain and upgrade to meet the needs of future data scale and technological development.

[0078] In a big data environment, duplicate data records not only take up additional storage space, but also increase the complexity of data processing and the consumption of computing resources. In data analysis and decision support, duplicate data records may lead to deviations and inaccuracies in analysis results. In addition, duplicate data records involving privacy and security will increase the risk of exposure of sensitive information and reduce data security.

[0079] The purpose of this invention is to provide important support for efficient data processing and analysis in a big data environment by eliminating data redundancy, optimizing resource utilization, improving the accuracy and real-time performance of data analysis, ensuring data consistency and integrity, and enhancing data security.

[0080] like Figure 1 FIG. 1 is a flowchart of a method for real-time data deduplication based on Flink provided by an embodiment of the present invention, referring to FIG. Figure 1 The embodiment of the present invention provides a real-time data deduplication method based on Flink, which specifically includes the following steps:

[0081] S101. Determine deduplication rules and control strategies, and submit Flink tasks.

[0082] S102, obtaining real-time data in the message queue as an input data stream, and determining a key value of the input data stream according to a deduplication rule, and then grouping the input data stream according to the key value through a grouping operator to obtain a grouped data stream;

[0083] S103, performing data deduplication on the packet data stream according to the key value by using a deduplication operator to obtain a main data stream and a side stream;

[0084] S104: Output the main data stream, and determine whether to save the side stream according to the control strategy.

[0085] Specifically, Figure 2 The figure shows a specific flow chart of the real-time data deduplication method based on Flink provided by an embodiment of the present invention. The embodiment of the present invention determines the key value of the input data stream according to the deduplication rule, groups the input data stream according to the key value through the grouping operator, and deduplicates the grouped data stream according to the key value through the deduplication operator. Real-time data deduplication is implemented based on Flink, and the efficiency and reliability of data deduplication are improved.

[0086] As a further optional implementation, the deduplication rule includes several data types and corresponding deduplication fields, and the control strategy includes control parameters and storage strategy.

[0087] Specifically, the embodiment of the present invention submits a Flink task after specifying a configuration file of a deduplication rule and a control strategy, wherein:

[0088] Deduplication rules are used to record deduplication field sets of different types of data, such as Figure 3 The figure shows a schematic diagram of a deduplication rule provided by an embodiment of the present invention, wherein the data types of the record include 001 and 002, and the deduplication fields of the record include field1, field2, field3, and field4. Specifically, for the data type 001, the corresponding deduplication fields are field1 and field2, and for the data type 002, the corresponding deduplication fields are field1, field2, field3, and field4;

[0089] The control strategy is used to record the parameter set and storage strategy (diversion strategy) that control the deduplication logic, such as Figure 4 The figure shows a schematic diagram of a control strategy provided by an embodiment of the present invention. The control parameters include window size, sliding step and expected number. The window size is the size of the time window, the value is 5, and the unit is minutes. The sliding step is the length of the time record for each adjustment of the linked list node, the value is 1, and the unit is minutes. The expected number is the amount of data expected to be inserted into the bit array, the value is 5000000. The storage strategy includes whether to shunt. Whether to shunt indicates whether redundant data is shunt and saved, which is set to true (yes).

[0090] Based on the Flink task, real-time data in the message queue is pulled as the input data stream, such as Figure 5The figure shows a schematic diagram of a message queue provided by an embodiment of the present invention. It can be seen that a plurality of real-time data are stored in sequence in the message queue, and each real-time data includes a data type, a timestamp and several field values.

[0091] The input data stream is grouped by the grouping operator according to the calculated key value.

[0092] like Figure 6 FIG. 1 is a flow chart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 6 As an optional implementation, the key value of the input data stream is determined according to the deduplication rule, which specifically includes:

[0093] S1021, determining the current data type of the input data stream, and determining the current deduplication field corresponding to the current data type according to the deduplication rule;

[0094] S1022, reading deduplication field values ​​of the input data stream according to the current deduplication field, and splicing the deduplication field values ​​to obtain a spliced ​​field value;

[0095] S1023. Determine a key value according to a hash value of the concatenated field value.

[0096] like Figure 7 FIG. 1 is another step flow chart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 7 As an optional implementation, the input data stream is grouped according to the key value by a grouping operator to obtain a grouped data stream, which specifically includes:

[0097] S1024. Implement Flink's KeySelector interface based on the grouping operator;

[0098] S1025. Divide a number of input data streams with the same key value into a group to obtain grouped data streams corresponding to the key value.

[0099] Specifically, the grouping operator implements Flink's KeySelector interface; reads the deduplication fields that match the current data type in the deduplication rules, concatenates the values ​​of all deduplication fields from the data stream, and calculates their md5 hash values ​​as the grouping key (i.e., key value) required by the grouping operator. The grouping operator can then group data according to the key.

[0100] like Figure 8 FIG. 1 is a schematic diagram of data grouping before and after the data grouping according to an embodiment of the present invention. It can be seen that the data type of the current input data stream is 001, and the deduplication fields of the corresponding data types in the deduplication rule are obtained, namely field1 and field2, and then the deduplication field values ​​corresponding to the deduplication fields are determined in the current input data stream. Figure 8 For the first and third rows of data, the duplicated field values ​​are value1_1 and value1_2. Figure 8 For the second row of data, the duplicate field values ​​are value2_1 and value2_2, value1_1 and value1_2 are concatenated to obtain the concatenated field value value1_1+value1_2, and the md5 hash value is calculated to obtain key1. Value2_1 and value2_2 are concatenated to obtain the concatenated field value value2_1+value2_2, and the md5 hash value is calculated to obtain key2. The grouping operator divides the input data streams with the same key value into a group, and the grouped data streams corresponding to key1 and key2 can be obtained.

[0101] like Fig. 9 FIG. 1 is a flow chart of step S103 provided in an embodiment of the present invention, referring to FIG. Fig. 9 As an optional implementation, the packet data stream is deduplicated according to the key value by using a deduplication operator to obtain a main data stream and a side stream, which specifically includes:

[0102] S1031, inherit Flink's KeyedProcessFunction abstract class through the deduplication operator;

[0103] S1032, determining the time window and sliding step size according to the control parameters of the control strategy, and creating a linked list, and then encapsulating the bit array and time record through the nodes of the linked list;

[0104] S1033, reading the packet data stream through the time window, and determining whether the current key value corresponding to the current data stream of the current time window exists in the linked list;

[0105] S1034. If the current key value does not exist in the linked list, collect the current data stream into the main data stream and add the current key value to the node of the linked list; if the current key value exists in the linked list, collect the current data stream into the side stream;

[0106] S1035. Update the current time window according to the sliding step size, and return to the step of determining whether the current key value corresponding to the current data stream of the current time window exists in the linked list.

[0107] Specifically, the grouped data flows to the deduplication operator to deduplicate data according to the key, and the non-duplicate data is output to the main data stream, and the duplicate data is output to the side stream. Fig.10A schematic diagram of data deduplication before and after the embodiment of the present invention is provided. It can be seen that for the two packet data streams corresponding to key1, since their key values ​​are the same, after deduplication processing by the deduplication operator, one packet data stream is saved in the main data stream, and the other packet data stream is output to the side stream. The specific process of data deduplication is as follows:

[0108] 1) The deduplication operator inherits Flink's KeyedProcessFunction abstract class;

[0109] 2) According to the parameters in the control strategy, a time window with a sliding step is maintained, and the deduplication operator only processes the data within the window; then a linked list is maintained, and the nodes of the linked list encapsulate a bit array and a time record;

[0110] 3) After receiving the packet data stream, the deduplication operator determines whether the key of the current data exists in the linked list. If not, the data is collected to the main data stream and the key of the data is added to the linked list; if so, the data is duplicated and the data is collected to the side stream.

[0111] like Fig.11 FIG. 1 is a flow chart of step S1034 provided in an embodiment of the present invention, referring to FIG. Fig.11 As an optional implementation, the current key value is added to the node of the linked list, which specifically includes:

[0112] S10341. When the linked list is empty, map the current key value to the bit array of the first node of the linked list, and update the time record of the first node according to the time and sliding step of the current data stream;

[0113] S10342, when the linked list is not empty, determine whether the time of the current data stream is less than the time record of the tail node of the linked list;

[0114] S10343. If the time of the current data flow is less than or equal to the time record of the tail node of the linked list, map the current key value to the bit array of the tail node of the linked list; if the time of the current data flow is greater than the time record of the tail node of the linked list, map the current key value to the newly added node of the linked list, and determine the time record of the newly added node based on the time record of the tail node and the sliding step.

[0115] Specifically, Fig.12 FIG. 1 is a schematic diagram of processing data streams based on time windows and linked lists according to an embodiment of the present invention, referring to FIG. Fig.12 , the logic of adding the key of the data to the linked list is as follows:

[0116] 1) If the linked list is empty, add the key to the new node (calculate multiple hash values ​​of the key and map them to the bit array in the node), and update the time record of the node to the time of the current data plus the sliding step parameter value;

[0117] 2) If the linked list is not empty, determine whether the time of the current data is less than or equal to the time record of the tail node of the linked list. If so, add the key to the tail node of the linked list;

[0118] 3) If the linked list is not empty and the time of the current data is greater than the time record of the tail node of the linked list, the key is added to the new node of the linked list. The time record of the new node is the time record of the original tail node plus the sliding step of the time window.

[0119] like Fig.13 The figure shows a schematic diagram of the bit array mapping of the linked list nodes in the time window provided by an embodiment of the present invention. The length of the bit array can be set in advance. When the key of the data is added to the linked list node, the hash value is calculated by hash operation and mapped to the bit array in the node, so as to facilitate the subsequent judgment based on the bit array whether the current key value corresponding to the current data stream exists in the linked list.

[0120] like Fig.14 FIG. 1 is a flow chart of step S1033 provided in an embodiment of the present invention, referring to FIG. Fig.11 As an optional implementation, it is further determined whether the current key value corresponding to the current data stream of the current time window exists in the linked list, which specifically includes:

[0121] S10331, traverse all nodes of the linked list, and calculate the mapping subscript of the current key value at each position in the bit array of each node;

[0122] S10332. When the values ​​of the mapping subscripts of the current key value at each position in the bit array of a certain node are all 1, it is determined that the current key value exists in the linked list.

[0123] Specifically, all nodes of the linked list are traversed, and the mapping subscripts of the hash value of the current key to each position in the bit array of each node are calculated. If the values ​​of all subscript positions of the bit array of a node are 1, it means that the current key value corresponding to the current data stream in the current time window exists in the linked list, that is, the current data stream is repeated.

[0124] In some optional embodiments, the embodiments of the present invention further include maintenance of the time window, as follows:

[0125] Create a new thread to maintain the time window, and check the time record of the first node of the linked list by comparing it with the current system time. If it exceeds the range of the time window, delete the first node (that is, the node that exceeds the time window will no longer participate in data deduplication).

[0126] In some optional embodiments, after obtaining the main data stream and the side stream, the main data stream data is output to the message queue, and according to the storage strategy, it is determined whether the side stream (redundant data) needs to be output to the message queue.

[0127] The above is a description of the method flow of the embodiment of the present invention. The following is a further description of the implementation process of the embodiment of the present invention in conjunction with a specific embodiment.

[0128] First, configure the message queue, deduplication rules, and control strategy in the preparation phase.

[0129] Message queue: Kafka that stores duplicate data.

[0130] Deduplication rules: Configuration files that contain deduplication field rules for different types of data.

[0131] Control strategy: A configuration file that contains the control parameters required by the program.

[0132] Then enter the data processing stage, such as Fig.15 FIG. 1 is a data interaction diagram of a real-time data deduplication method based on Flink provided by an embodiment of the present invention, referring to FIG. Fig.15 , the process of data processing stage is as follows:

[0133] 1) The Flink Source reads the data in the message queue (specifically the input Topic in Kakfa) as the input stream;

[0134] 2) The grouping operator receives the input stream and reads the rule that matches the current data type in the deduplication rule to construct a grouping key and group the data. After grouping, the data stream is returned as a grouped stream;

[0135] 3) The deduplication operator receives the packet stream and determines whether the data is duplicated based on the time window, linked list plus bit array data structure constructed by the control parameters, and collects the data into the main data stream or side stream based on the judgment result;

[0136] 4) The main data stream data is output to the message queue configured on the Flink Sink side (specifically the Topic storing the main data stream in Kafka) for storage. The side stream data is determined whether to be stored in the message queue (specifically the Topic storing the side stream data in Kafka) and executed based on the configuration of the storage strategy.

[0137] It can be understood that the embodiment of the present invention uses Flink's custom grouping function (grouping operator) and custom processing function (deduplication operator) based on two data structures, linked list and bit array, to achieve deduplication of data within the time window range, and provide technical support for eliminating data redundancy in a big data environment, optimizing resource utilization, improving the accuracy and real-time performance of data analysis, and enhancing data security. While taking into account the traditional deduplication scheme, the present invention avoids its shortcomings and takes into account the advantages of real-time performance, scalability, flexibility, low complexity, fast query, and low resource consumption. The embodiment of the present invention determines the key value of the input data stream according to the deduplication rule, groups the input data stream according to the key value through the grouping operator, and performs data deduplication on the grouped data stream according to the key value through the deduplication operator, and realizes real-time data deduplication based on Flink, thereby improving the efficiency and reliability of data deduplication.

[0138] Compared with the prior art, the embodiments of the present invention also have the following advantages:

[0139] 1) Efficient space utilization and fast query: By adopting an optimized combination of linked lists and bit arrays, our solution achieves efficient storage and retrieval of data, avoiding the high space complexity of hash tables while maintaining fast query capabilities, significantly improving data processing efficiency.

[0140] 2) Simplify system architecture and improve real-time performance: The present invention is implemented by customizing the Flink deduplication operator. This measure greatly simplifies the data processing process, reduces dependence on external components, eliminates the delay caused by network communication, enhances the real-time response capability of the solution, and meets the stringent speed requirements of modern data processing.

[0141] 3) Flexible and customizable, resource consumption optimization: Based on Flink's implementation, the present invention not only retains the flexibility and customizability of feature engineering, but also further optimizes resource consumption and lowers the technical threshold, making the solution easy to deploy and maintain while ensuring real-time performance, and adapting to the processing requirements of data sets of different sizes.

[0142] 4) High-precision deduplication without misjudgment: The present invention realizes high-precision data deduplication without misjudgment, and at the same time has high scalability and fault tolerance, providing reliable guarantee for accurate data processing and management.

[0143] 5) Comprehensively optimized performance: The present invention comprehensively considers multiple dimensions such as real-time performance, scalability, flexibility, low complexity, fast query and resource consumption, and forms a fully optimized data deduplication solution, which not only effectively solves the limitations of traditional methods, but also brings innovative solutions to the field of big data processing.

[0144] The real-time data deduplication method of the present invention greatly reduces the implementation difficulty and resource consumption on the basis of ensuring data processing efficiency and accuracy, provides strong support for data management and analysis in the big data era, and has significant innovative value and practical prospects.

[0145] like Fig.16 FIG. 1 is a schematic diagram of a real-time data deduplication device based on Flink provided in an embodiment of the present invention. Fig.16 , an embodiment of the present invention provides a real-time data deduplication device based on Flink, comprising:

[0146] The task submission module is used to determine deduplication rules and control strategies and submit Flink tasks;

[0147] The data grouping module is used to obtain the real-time data in the message queue as the input data stream, determine the key value of the input data stream according to the deduplication rule, and then group the input data stream according to the key value through the grouping operator to obtain the grouped data stream;

[0148] A data deduplication module is used to perform data deduplication on the packet data stream according to the key value through a deduplication operator to obtain a main data stream and a side stream;

[0149] The data output module is used to output the main data stream and determine whether to save the side stream according to the control strategy.

[0150] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] The embodiment of the present invention further provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned real-time data deduplication method based on Flink. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0152] like Fig.17 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention, referring to FIG. Fig.17 , an embodiment of the present invention provides an electronic device, including:

[0153] The processor 1701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0154] The memory 1702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1702 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1702, and the processor 1701 calls and executes the real-time data deduplication method based on Flink in the embodiment of the present invention;

[0155] Input / output interface 1703, used to implement information input and output;

[0156] Communication interface 1704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0157] A bus 1705 that transmits information between the various components of the device (e.g., the processor 1701, the memory 1702, the input / output interface 1703, and the communication interface 1704);

[0158] The processor 1701 , the memory 1702 , the input / output interface 1703 and the communication interface 1704 are connected to each other in communication within the device via the bus 1705 .

[0159] like Fig.18 FIG. 1 is a schematic diagram of a storage medium according to an embodiment of the present invention. Fig.18 An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs 1801, and the one or more programs 1801 can be executed by one or more processors to implement the above-mentioned real-time data deduplication method based on Flink.

[0160] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.

[0162] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0163] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0164] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0165] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0166] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the above-mentioned program is printed, since the above-mentioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in other suitable ways as necessary, and then stored in a computer memory.

[0167] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0168] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0169] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0170] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A real-time data deduplication method based on Flink, characterized in that: The following steps are involved: Determine deduplication rules and control strategies, and submit Flink tasks; Acquire real-time data in the message queue as an input data stream, determine a key value of the input data stream according to the deduplication rule, and then group the input data stream according to the key value through a grouping operator to obtain a grouped data stream; Deduplication is performed on the packet data stream according to the key value by using a deduplication operator to obtain a main data stream and a side stream; The main data stream is output, and whether to save the side stream is determined according to the control strategy.

2. According to the Flink-based real-time data deduplication method of claim 1, it is characterized by: The deduplication rules include several data types and corresponding deduplication fields, and the control strategy includes control parameters and storage strategy.

3. According to the real-time data deduplication method based on Flink according to claim 1, it is characterized in that: The step of determining the key value of the input data stream according to the deduplication rule specifically includes: Determine a current data type of the input data stream, and determine a current deduplication field corresponding to the current data type according to the deduplication rule; Reading a deduplication field value of the input data stream according to the current deduplication field, and splicing the deduplication field value to obtain a spliced ​​field value; The key value is determined according to the hash value of the concatenated field value.

4. According to a Flink-based real-time data deduplication method according to claim 1, it is characterized in that: The step of grouping the input data stream according to the key value by using a grouping operator to obtain a grouped data stream specifically includes: Implement Flink's KeySelector interface based on the grouping operator; Several input data streams with the same key value are divided into a group to obtain the grouped data streams corresponding to the key value.

5. According to a Flink-based real-time data deduplication method according to claim 1, it is characterized in that: The deduplication operator is used to perform data deduplication on the packet data stream according to the key value to obtain a main data stream and a side stream, which specifically includes: Inherit Flink's KeyedProcessFunction abstract class through the deduplication operator; Determine the time window and the sliding step according to the control parameters of the control strategy, create a linked list, and then encapsulate the bit array and the time record through the nodes of the linked list; Reading the packet data stream through the time window, and determining whether a current key value corresponding to a current data stream in a current time window exists in the linked list; If the current key value does not exist in the linked list, collect the current data stream into the main data stream and add the current key value to the node of the linked list; if the current key value exists in the linked list, collect the current data stream into the side stream; The current time window is updated according to the sliding step, and the step of returning to determine whether the current key value corresponding to the current data stream of the current time window exists in the linked list is returned.

6. According to the real-time data deduplication method based on Flink according to claim 5, it is characterized in that: The step of adding the current key value to the node of the linked list specifically includes: When the linked list is empty, the current key value is mapped to the bit array of the first node of the linked list, and the time record of the first node is updated according to the time of the current data stream and the sliding step; When the linked list is not empty, determining whether the time of the current data stream is less than the time record of the tail node of the linked list; If the time of the current data flow is less than or equal to the time record of the tail node of the linked list, the current key value is mapped to the bit array of the tail node of the linked list; if the time of the current data flow is greater than the time record of the tail node of the linked list, the current key value is mapped to a newly added node of the linked list, and the time record of the newly added node is determined based on the time record of the tail node and the sliding step.

7. The real-time data deduplication method based on Flink according to claim 5 is characterized in that: The determining whether the current key value corresponding to the current data stream of the current time window exists in the linked list specifically includes: Traversing all nodes of the linked list, calculating the mapping subscript of the current key value at each position in the bit array of each node; When the values ​​of the mapping subscripts of the current key value at each position in the bit array of a certain node are all 1, it is determined that the current key value exists in the linked list.

8. A real-time data deduplication device based on Flink, characterized in that: include: The task submission module is used to determine deduplication rules and control strategies and submit Flink tasks; A data grouping module, used to obtain real-time data in the message queue as an input data stream, and determine a key value of the input data stream according to the deduplication rule, and then group the input data stream according to the key value through a grouping operator to obtain a grouped data stream; A data deduplication module, configured to perform data deduplication on the packet data stream according to the key value by using a deduplication operator to obtain a main data stream and a side stream; A data output module is used to output the main data stream and determine whether to save the side stream according to the control strategy.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the real-time data deduplication method based on Flink are realized as described in any one of claims 1 to 7.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the real-time data deduplication method based on Flink as described in any one of claims 1 to 7.

Images