A data deduplication method, apparatus, computer equipment, and storage medium

Abstract

This application discloses a data deduplication method, apparatus, computer device, and storage medium in the field of communication technology. The method can acquire log data of a target application, the log data including device numbers of multiple user devices; map each device number in the log data to a numerical sequence number to construct a global dictionary of device sequence numbers, where the device number and the numerical sequence number have a one-to-one mapping relationship; construct a device deduplication lookup table based on the log data and the device numbers; when a device deduplication request from a target user device is received, a target numerical sequence number is determined based on the device deduplication request, the device deduplication lookup table, and the global dictionary of device sequence numbers; deduplication is performed on the target numerical sequence number to obtain the deduplication result corresponding to the device deduplication request. This application can improve the speed of data deduplication.

Description

Technical Field

[0001] This application relates to the field of communication technology, specifically to a data deduplication method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of communication technology, deduplication of user devices (or accounts) is widely used in daily ad hoc analytics, such as calculating metrics like Daily Active Users, membership numbers, conversion rates, and retention rates.

[0003] In the process of researching and practicing related technologies, the inventors of this application discovered that deduplication and counting of massive amounts of data is very resource-intensive and is one of the biggest bottlenecks to improving query speed. When performing deduplication operations on big data, the original values ​​of key-value pairs distributed across different nodes need to be aggregated into the same node through a shuffle function, which consumes a lot of disk and network I / O. Furthermore, numerical comparison operations are required during search and insertion, resulting in high computational complexity. Summary of the Invention

[0004] This application provides a data deduplication method, apparatus, device, and medium to solve the technical problem that when performing deduplication on big data, numerical comparison operations are required, resulting in high computational complexity and slow deduplication speed, which in turn affects the speed of ad-hoc queries.

[0005] In a first aspect, embodiments of this application provide a data deduplication method, including:

[0006] Obtain log data of the target application, wherein the log data includes device IDs of multiple user devices;

[0007] Each device number in the log data is mapped to a numerical sequence number to construct a global dictionary of device sequence numbers, wherein the device number and the numerical sequence number are in a one-to-one mapping relationship.

[0008] Based on the log data and the device number, a device deduplication query table is constructed;

[0009] When a device deduplication request for a target user device is received, a target numerical sequence number is determined based on the device deduplication request, the device deduplication lookup table, and the device serial number global dictionary. Deduplication is performed on the target numerical sequence number to obtain the deduplication result corresponding to the device deduplication request.

[0010] Optionally, determining the target numerical sequence number based on the equipment deduplication request, the equipment deduplication lookup table, and the equipment sequence number global dictionary includes:

[0011] Based on the equipment deduplication request, the equipment number that needs to be deduplicated is determined from the equipment deduplication lookup table;

[0012] Obtain the target numeric sequence number corresponding to the device number that needs to be deduplicated from the global dictionary of device serial numbers.

[0013] Optionally, deduplication is performed on the target numerical sequence to obtain the deduplication result corresponding to the device deduplication request, including:

[0014] Convert the target number sequence into a binary target bit array;

[0015] The target bit array is used as input parameter, and the cardinality of the target bit array is calculated by the preset groupBitmap function. The cardinality is the deduplication result corresponding to the device deduplication request.

[0016] Optionally, a global dictionary of device serial numbers is constructed by mapping each device number in the log data to a numerical sequence number, including:

[0017] The device numbers in the log data are sorted according to a preset order to obtain a sorted set of device numbers;

[0018] Map each device number in the sorted device number set to a numerical sequence number to obtain a global dictionary of device sequence numbers.

[0019] Optionally, a global dictionary of device serial numbers is constructed by mapping each device number in the log data to a numerical sequence number, including:

[0020] Obtain log data within a preset initial time partition, wherein the log data within the initial time partition includes the initial partition device number;

[0021] Map each of the initial partition device numbers to a numerical sequence number to construct a global dictionary of device sequence numbers;

[0022] If log data indicating a device number update is received, then the global dictionary of device serial numbers is updated.

[0023] Optionally, if log data indicating a device number update is received, the global dictionary of device serial numbers is updated, including:

[0024] If log data with updated device number is received, the device number corresponding to the updated device number is mapped one by one to a numeric sequence number to obtain a set of numeric sequence numbers;

[0025] Update the global dictionary of device serial numbers based on the set of numerical serial numbers.

[0026] Optionally, based on the set of numerical serial numbers, the global dictionary of device serial numbers is updated, including:

[0027] Match the numerical serial numbers in the global dictionary of device serial numbers with the numerical serial numbers in the set of device serial numbers;

[0028] Add any mismatched numeric serial numbers to the device serial number global dictionary.

[0029] Secondly, embodiments of this application provide a data deduplication device, comprising:

[0030] The acquisition unit is used to acquire log data of the target application, wherein the log data includes device numbers of multiple user devices;

[0031] A dictionary construction unit is used to map each device number in the log data to a numerical sequence number to construct a global dictionary of device numbers, wherein the device number and the numerical sequence number are in a one-to-one mapping relationship.

[0032] The query table construction unit is used to construct a device deduplication query table based on the log data and the device number;

[0033] The deduplication unit is used to, when receiving a device deduplication request from a target user device, determine a target numerical sequence number based on the device deduplication request, the device deduplication lookup table, and the device sequence number global dictionary, perform deduplication on the target numerical sequence number, and obtain the deduplication result corresponding to the device deduplication request.

[0034] Thirdly, embodiments of this application also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the steps in any of the data deduplication methods provided in embodiments of this application.

[0035] Fourthly, embodiments of this application also provide a storage medium storing a plurality of instructions, which are applicable to a processor for loading to execute the steps in any of the data deduplication methods provided in embodiments of this application.

[0036] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application can obtain log data of a target application, wherein the log data includes device numbers of multiple user devices; it maps each device number in the log data to a numerical sequence number, thereby constructing a global dictionary of device sequence numbers, wherein the device number and the numerical sequence number have a one-to-one mapping relationship; it constructs a device deduplication lookup table based on the log data and the device number; when a device deduplication request from a target user device is received, it uses the device deduplication request, the device deduplication lookup table, and the global dictionary of device sequence numbers to construct a global dictionary of device sequence numbers. The dictionary determines the target numeric sequence number, and deduplication is performed on the target numeric sequence number to obtain the deduplication result corresponding to the device deduplication request. The data deduplication method proposed in this application proposes a scheme to calculate the number of unique elements without using the original key-value pair information. In specific implementation, the device number can be mapped to a numeric sequence number to generate an incrementally updatable and unique global dictionary of device sequence numbers. The deduplication calculation is performed using the unsigned numeric sequence numbers in the global dictionary of device sequence numbers. This can reduce the computational complexity of the deduplication operation while occupying very little memory, thereby improving the data deduplication speed and thus improving the ad-hoc query speed. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0040] Figure 1 This is a flowchart of the data deduplication method provided in the embodiments of this application;

[0041] Figure 2 This is a system block diagram of the data deduplication method provided in the embodiments of this application;

[0042] Figure 3 This is a diagram of an apparatus for a data deduplication method provided in an embodiment of this application;

[0043] Figure 4This is another apparatus diagram of the data deduplication method provided in the embodiments of this application;

[0044] Figure 5 This is another apparatus diagram of the data deduplication method provided in the embodiments of this application;

[0045] Figure 6 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0048] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0049] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0050] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0051] To address the technical problem that existing technologies require numerical comparison operations when performing deduplication on large datasets, resulting in high computational complexity and slow deduplication speed, which in turn affects ad-hoc query speed, this application provides a data deduplication method that can reduce the computational complexity of deduplication operations while occupying minimal memory, thereby improving the speed of data deduplication and ultimately enhancing ad-hoc query speed.

[0052] Figure 1 A data deduplication method provided in this application embodiment includes:

[0053] S101. Obtain the log data of the target application.

[0054] The log data may include device numbers of multiple user devices.

[0055] In one example, the log data may include log data within a preset time period, which can be preset days, weeks, months, or other dates. For example, the log data may be the target application's log data within March 1, 2023 (one day), the log data may be the target application's log data from June 5, 2023 to June 11, 2023 (one week), the log data may also be the target application's log data from June 1, 2023 to June 30, 2023 (one month), and so on.

[0056] In one example, the startup log details of the target application are used as an example for illustration. The startup log details may contain information such as the device number and region of the user device corresponding to the target application. After obtaining the startup log details, the device number of the user device of the target application can be obtained.

[0057] In one example, other business data can also be obtained, and a dictionary can be built through the mapping between the data to form a common data-level service. All teams use a common dictionary, so that when users query data produced across teams, they can arbitrarily combine data tables to query data.

[0058] S102: Map each device number in the log data to a numerical sequence number to construct a global dictionary of device sequence numbers.

[0059] The device number and the numerical sequence number are in a one-to-one mapping relationship. The device number in the log data can be mapped to the numerical sequence number one by one. The one-to-one mapping relationship between the device number and the numerical sequence number is stored as a global dictionary of device sequence numbers.

[0060] The device serial number global dictionary can be appended, and it guarantees a unique mapping between device numbers and numerical serial numbers at each time period.

[0061] In one example, when the log data is within a preset time period (one day), the device number corresponding to the log data of the target application within that day is mapped to a numeric sequence number. When log data updated with device numbers from other dates is received, the device numbers of the log data from other dates are mapped to numeric sequence numbers and appended to the previous global device number dictionary. The device numbers and numeric sequence numbers in the global device number dictionary are in a one-to-one correspondence.

[0062] This application embodiment constructs a global dictionary of device serial numbers that maps device numbers to numerical serial numbers one-to-one. When performing data deduplication calculations, each device number corresponds to a unique numerical serial number, eliminating the need for repeated calculations. This improves the speed of data deduplication and, consequently, the speed of ad-hoc queries.

[0063] In one embodiment, the step "mapping each device number in the log data with a numerical sequence number to construct a global dictionary of device sequence numbers" may include:

[0064] The device numbers in the log data are sorted according to a preset order to obtain a sorted set of device numbers;

[0065] Map each device number in the sorted device number set to a numerical sequence number to obtain a global dictionary of device sequence numbers.

[0066] In one example, device numbers can be sorted from 0 in any way so that each device number can be uniquely mapped to an unsigned integer sequence number, such as device A being mapped to data sequence number 0 and device B being mapped to numeric sequence number 1.

[0067] In one example, data can be deduplicated using a Bitmap, but Bitmap deduplicatizes unsigned numbers. Therefore, device numbers can be mapped one by one to data sequence numbers before performing data deduplication.

[0068] This can be achieved by defining a bit array, where each element corresponds to a single bit in the array. For example, consider a set of device numbers mapped to the number sequence [2, 3, 5, 7], where device number A maps to number 2, device number B to number 3, device number C to number 5, and device number D to number 7. The corresponding bit array using these indices is [00110101], which is a bitmap. The bitmap only needs to use 0 and 1 values; it does not need to store key-value pairs.

[0069] In this application, the subscript refers to the subscript of the bit array. For example, if an element is 2, the value of the third bit in the corresponding bit array is 1 (1 represents the element exists, 0 represents the element does not exist, and the subscript starts from 0), represented as:

[001] . The array subscript must be a positive integer, so the device number needs to be mapped to an integer for deduplication. Searching and insertion operations in the bitmap only require bitwise operations, which have low computational complexity and are faster. In principle, this reduces network consumption and computational complexity, greatly improving the speed of deduplication calculation.

[0070] Bitwise operations refer to operations performed on binary bits (AND, OR, XOR, etc.). A bitmap stores a binary array, and bitwise operations are used when performing deduplication calculations on the bitmap. For example, if the device number of device A is mapped to a number 2, and the device number of device B is mapped to a number 0, then during the deduplication calculation using the bitmap, the number 2 can be represented as

[001] , and the number 0 can be represented as

[100] . By performing an OR operation on the bit arrays stored in the two devices, that is, by calculating

[001] and

[100] , we can obtain

[101] . Then, by calculating the base of the array, that is, by calculating the number of 1s in

[101] , we get 2, and thus the deduplication count is 2.

[0071] Furthermore, since sparse and dense bitmaps occupy the same amount of space, the smaller the number of bits in the bitmap, the better. Therefore, it is necessary to use an index mapping method to compress the storage space of the bitmap to the greatest extent possible and avoid excessive memory consumption.

[0072] In one embodiment, the step "mapping each device number in the log data with a numerical sequence number to construct a global dictionary of device sequence numbers" may include:

[0073] Obtain log data within a preset initial time partition, wherein the log data within the initial time partition includes the initial partition device number;

[0074] Map each of the initial partition device numbers to a numerical sequence number to construct a global dictionary of device sequence numbers;

[0075] If log data indicating a device number update is received, then the global dictionary of device serial numbers is updated.

[0076] In one example, when deduplication of data across time partitions is required, the log data in the initial time partition can be obtained initially. All device numbers in the log data in the initial time partition can be sorted from 0 in any way to obtain corresponding one-to-one mapping unsigned integers (numeric serial numbers), which are saved as a global dictionary of device serial numbers. This global dictionary of device serial numbers can be appended and is guaranteed to be uniquely mapped across all time partitions.

[0077] Furthermore, when writing to a subsequent time partition, the serial number mapping of new devices not in the device serial number dictionary within the time partition must be incrementally updated. Devices not in the dictionary are defined as new devices. For example, if the largest numerical serial number of an existing device in the dictionary is 3, and two unknown devices not in the device serial number dictionary are added, their numerical serial numbers could be 4 and 5. In this way, all devices within the newly written time partition can be associated with an integer serial number.

[0078] In one embodiment, the step "if log data of device number update is received, then update the device serial number global dictionary" may include:

[0079] If log data with updated device number is received, the device number corresponding to the updated device number is mapped one by one to a numeric sequence number to obtain a set of numeric sequence numbers;

[0080] Update the global dictionary of device serial numbers based on the set of numerical serial numbers.

[0081] In one embodiment, the step "update the device serial number global dictionary based on the set of numerical serial numbers" may include:

[0082] Match the numerical serial numbers in the global dictionary of device serial numbers with the numerical serial numbers in the set of device serial numbers;

[0083] Add any mismatched numeric serial numbers to the device serial number global dictionary.

[0084] Here, mismatched serial numbers refer to serial numbers that do not match the serial numbers in the device serial number set and the serial numbers in the device serial number global dictionary. They can also be understood as serial numbers that exist in the device data serial number set but do not exist in the device serial number global dictionary.

[0085] S103. Based on log data and device number, construct the device deduplication query table.

[0086] In one embodiment, log data and device numbers can be stored in the ClickHouse columnar database management system to construct a device deduplication lookup table.

[0087] The deduplication calculation is ad-hoc, storing detailed data and device serial numbers in a table within ClickHouse. Users can query the detailed data in the table using any custom conditions and calculate the results in real time.

[0088] In one embodiment, log data and mapped numerical serial numbers can be stored as a table in ClickHouse. Users can query detailed data in the table using custom conditions and calculate results in real time. Building a global dictionary of device serial numbers is primarily to ensure the uniqueness of mappings for deduplicated data across partitions (days). For example, if device A existed yesterday with a mapped numerical serial number of 1, and device A also exists today, its numerical serial number needs to be retrieved from the global device dictionary as 1.

[0089] In one example, the device number in the log data is associated with a global dictionary of device serial numbers. The log data only adds a dimension of numeric serial numbers corresponding to the device number. This log data with the added numeric serial number dimension can be stored in ClickHouse. Subsequent queries still retrieve the log data, but deduplication is performed using the numeric serial numbers corresponding to the device serial numbers.

[0090] In one example, the startup log details of the target application can be processed and stored in ClickHouse. Users can customize conditions to query the number of daily active users under different conditions for analysis, such as querying the number of active devices in Beijing, the number of active devices on Android and Apple, etc.

[0091] In one example, such as Figure 2 As shown, when cross-partition deduplication is not required, it is only necessary to map device numbers one-to-one to numeric serial numbers, then build a global dictionary of device serial numbers, and store log data and device numbers as a table in ClickHouse. Users can use custom conditions to query detailed data in the table and calculate results in real time. When cross-partition deduplication is required, during the initial calculation, all device numbers are sorted from 0 in any way to obtain corresponding unsigned integers with one-to-one mappings, and this is saved as a global dictionary of device serial numbers. This global dictionary of device serial numbers can be appended and guarantees unique mapping across all partitions. When subsequent partitions are written, the serial number mapping of new devices in the partition that are not in the global dictionary of device serial numbers must be incrementally updated. New devices can be identified by the global dictionary of device serial numbers on day T-1. Devices not in the global dictionary of device serial numbers are defined as new devices. For example, if the largest numeric serial number of an existing device in the dictionary is 3, and two unknown devices not in the dictionary are added, their numeric serial numbers are 4 and 5. All devices in the newly written partition can be associated with an integer serial number.

[0092] In this context, "partition" refers to the periodic partitioning of log data calculations, typically based on dates such as days, weeks, or months. For example, if data is generated daily, the partition would be the current day.

[0093] Where T can represent the current production date, T-1 is the previous production date, and the global dictionary of equipment serial numbers for T-1 refers to the global dictionary of equipment serial numbers calculated in the previous cycle. Equipment not in the global dictionary of equipment serial numbers is defined as new equipment.

[0094] This application embodiment constructs a device deduplication lookup table that combines log data with device numbers or log data with numerical serial numbers. When users subsequently query data (data deduplication calculation), they can directly query through the device deduplication lookup table, thereby improving the speed of ad-hoc queries.

[0095] S104. When a device deduplication request for a target user device is received, the target numerical sequence number is determined based on the device deduplication request, the device deduplication lookup table, and the global dictionary of device serial numbers. The target numerical sequence number is deduplicated to obtain the deduplication result corresponding to the device deduplication request.

[0096] The target number is a number from the global dictionary of device numbers. The target number can be used to deduplicate and obtain the deduplication result.

[0097] In one example, the device deduplication request for the target user device can include a deduplication calculation period. The device number within the deduplication calculation period can be obtained from the device deduplication lookup table. Since the device serial number global dictionary stores a one-to-one mapping relationship between device numbers and numeric serial numbers, the corresponding numeric serial number, i.e., the target numeric serial number, can be found from the device serial number global dictionary through the device number. Then, the deduplication result can be obtained by performing deduplication on the target numeric serial number.

[0098] In this application embodiment, the original method of deduplication by device number is changed to deduplication by using the numeric serial number in the global dictionary of device serial numbers. This numeric serial number is an unsigned numeric serial number. Using this numeric serial number for deduplication calculation can reduce the complexity of deduplication calculation.

[0099] In one embodiment, the step "determining the target numerical sequence number based on the device deduplication request, the device deduplication lookup table, and the device sequence number global dictionary" may include:

[0100] Based on the equipment deduplication request, the equipment number that needs to be deduplicated is determined from the equipment deduplication lookup table;

[0101] Obtain the target numeric sequence number corresponding to the device number that needs to be deduplicated from the global dictionary of device serial numbers.

[0102] In one embodiment, the step "deduplicating the target numerical sequence to obtain the deduplication result corresponding to the device deduplication request" may include:

[0103] Convert the target number sequence into a binary target bit array;

[0104] The target bit array is used as input parameter, and the cardinality of the target bit array is calculated by the preset groupBitmap function. The cardinality is the deduplication result corresponding to the device deduplication request.

[0105] The input parameter is an unsigned integer (numeric index). The groupBitmap function can be used to return the deduplication value of these unsigned integers. Using the index of the deduplication device, the deduplication result can be obtained.

[0106] In one example, when performing deduplication calculations on devices, the `groupBitmap` function is used to deduplicat the numeric indices corresponding to the device IDs. The principle of `groupBitmap` is as follows: Assuming there are device IDs A, B, and D, which are mapped to 0, 1, and 3 respectively in a dictionary, these three numeric indices are used to generate a bit array [1, 1, 0, 1]. Calculating the radix of the array, i.e., the number of 1s, yields the deduplicated device ID count of 3. For instance, if a user needs to query the number of active devices in Beijing, using the deduplication method of this application, a deduplication result of 3 indicates that there are 3 active devices in Beijing.

[0107] As can be seen from the above, the data deduplication method proposed in this application provides a solution for calculating the number of unique elements without using the original key-value pair information. In specific implementation, a unique global dictionary of device serial numbers can be generated by mapping device numbers to numeric serial numbers. Deduplication calculations are then performed using the unsigned numeric serial numbers in the global dictionary of device serial numbers. This reduces the computational complexity of deduplication operations while occupying very little memory, thereby improving the speed of data deduplication and, consequently, the speed of ad-hoc queries.

[0108] To better implement the above methods, this application also provides a data deduplication device, which can be integrated into a computer device. (See reference...) Figure 3 The data deduplication device may include an acquisition unit 201, a dictionary construction unit 202, a lookup table construction unit 203, and a deduplication unit 204, as follows:

[0109] (1) Obtain unit 201;

[0110] The acquisition unit 201 is used to acquire log data of the target application, wherein the log data includes device numbers of multiple user devices.

[0111] (2) Dictionary construction unit 202;

[0112] The dictionary construction unit 202 is used to map each device number in the log data to a numerical sequence number to construct a global dictionary of device numbers, wherein the device number and the numerical sequence number are in a one-to-one mapping relationship.

[0113] In one embodiment, such as Figure 4 As shown, the dictionary construction unit 202 includes:

[0114] The sorting subunit 2021 is used to sort the device numbers in the log data according to a preset order to obtain a sorted device number set;

[0115] The first mapping subunit 2022 is used to map each device number in the sorted device number set to a numerical serial number to obtain a global dictionary of device serial numbers.

[0116] In one embodiment, such as Figure 4 As shown, the dictionary construction unit 202 includes:

[0117] The first acquisition subunit 2023 is used to acquire log data within a preset initial time partition, wherein the log data within the initial time partition includes the initial partition device number;

[0118] The second mapping subunit 2024 is used to map the initial partition device number to a numerical sequence number and construct a global dictionary of device sequence numbers.

[0119] The update subunit 2025 is used to update the global dictionary of device serial numbers if log data of device number update is received.

[0120] In one embodiment, the update subunit 2025 is further configured to, if receiving log data with updated device numbers, map the device numbers corresponding to the updated log data with updated device numbers to numeric serial numbers to obtain a set of numeric serial numbers; and update the global dictionary of device serial numbers based on the set of numeric serial numbers.

[0121] In one embodiment, the update subunit 2025 is further configured to match the numeric serial numbers in the device serial number global dictionary with the numeric serial numbers in the device serial number set; and to append the non-matching numeric serial numbers to the device serial number global dictionary.

[0122] (3) Query table construction unit 203;

[0123] The query table construction unit 203 is used to construct a device deduplication query table based on the log data and the device number.

[0124] (4) Weight-relief unit 204;

[0125] The deduplication unit 204 is used to, when receiving a device deduplication request from a target user device, determine a target numerical sequence number based on the device deduplication request, the device deduplication lookup table, and the device serial number global dictionary, perform deduplication on the target numerical sequence number, and obtain the deduplication result corresponding to the device deduplication request.

[0126] In one embodiment, such as Figure 5 As shown, the weight-removal unit 204 includes:

[0127] The determination subunit 2041 is used to determine the equipment number that needs to be deduplicated from the equipment deduplication query table based on the equipment deduplication request;

[0128] The second acquisition subunit 2042 is used to acquire the target numeric sequence number corresponding to the device number that needs to be deduplicated from the device sequence number global dictionary.

[0129] In one embodiment, such as Figure 5 As shown, the weight-removal unit 204 includes:

[0130] The conversion subunit 2043 is used to convert the target digital sequence number into a binary target bit array;

[0131] The calculation subunit 2044 is used to take the target bit array as input parameter and calculate the cardinality of the target bit array through the preset groupBitmap function. The cardinality is the deduplication result corresponding to the device deduplication request.

[0132] As can be seen from the above, the data deduplication device of this application embodiment acquires the log data of the target application, the log data including the device numbers of multiple user devices; then, the dictionary construction unit 202 maps each device number in the log data to a numerical sequence number to construct a global dictionary of device sequence numbers, the device number and the numerical sequence number having a one-to-one mapping relationship; the query table construction unit 203 constructs a device deduplication query table based on the log data and the device numbers; when the deduplication unit 204 receives a device deduplication request from the target user device, it determines the target numerical sequence number based on the device deduplication request, the device deduplication query table and the global dictionary of device sequence numbers, performs deduplication on the target numerical sequence number, and obtains the deduplication result corresponding to the device deduplication request.

[0133] The proposed data deduplication method offers a solution for calculating the number of unique elements without using the original key-value pair information. In practice, it can map device numbers to numeric serial numbers to generate an incrementally updatable, unique global dictionary of device serial numbers. Deduplication calculations are then performed using the unsigned numeric serial numbers within this global dictionary. This approach reduces the computational complexity of deduplication operations while consuming minimal memory, thereby increasing the speed of data deduplication and consequently improving ad-hoc query speed.

[0134] The aforementioned data de-weighting device can be implemented as a computer program, which can, for example... Figure 6 It runs on the computer device shown.

[0135] like Figure 6 As shown in the figure, this application provides a computer device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0136] Memory 113 is used to store computer programs;

[0137] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the data deduplication method provided in any of the foregoing method embodiments.

[0138] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0139] Therefore, this application also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the data deduplication method provided in any of the foregoing method embodiments.

[0140] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0141] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0143] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0145] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0146] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.

[0147] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data deduplication method, characterized in that, include: Obtain log data of the target application, wherein the log data includes device IDs of multiple user devices; Each device number in the log data is mapped to a numerical sequence number to construct a global dictionary of device sequence numbers. The device number and the numerical sequence number are in a one-to-one mapping relationship. The global dictionary of device sequence numbers can be appended. When log data updated by a device number is received, it is appended to the global dictionary of device sequence numbers. Based on the log data and the device number, a device deduplication query table is constructed; When a device deduplication request for a target user device is received, a target numeric sequence number is determined based on the device deduplication request, the device deduplication lookup table, and the device sequence number global dictionary. Deduplication is performed on the target numeric sequence number to obtain the deduplication result corresponding to the device deduplication request. This includes: converting the target numeric sequence number into a binary target bit array; using the target bit array as an input parameter, the cardinality of the target bit array is calculated through a preset groupBitmap function, where the cardinality is the deduplication result corresponding to the device deduplication request.

2. The method according to claim 1, characterized in that, The step of determining the target numerical sequence number based on the equipment deduplication request, the equipment deduplication lookup table, and the global dictionary of equipment serial numbers includes: Based on the equipment deduplication request, the equipment number that needs to be deduplicated is determined from the equipment deduplication lookup table; Obtain the target numeric sequence number corresponding to the device number that needs to be deduplicated from the global dictionary of device serial numbers.

3. The method according to claim 1, characterized in that, The process involves mapping each device number in the log data to a numerical sequence number, constructing a global dictionary of device sequence numbers, including: The device numbers in the log data are sorted according to a preset order to obtain a sorted set of device numbers; Map each device number in the sorted device number set to a numerical sequence number to obtain a global dictionary of device sequence numbers.

4. The method according to claim 1, characterized in that, The process involves mapping each device number in the log data to a numerical sequence number, constructing a global dictionary of device sequence numbers, including: Obtain log data within a preset initial time partition, wherein the log data within the initial time partition includes the initial partition device number; Map each of the initial partition device numbers to a numerical sequence number to construct a global dictionary of device sequence numbers; If log data indicating a device number update is received, then the global dictionary of device serial numbers is updated.

5. The method according to claim 4, characterized in that, If log data indicating a device number update is received, the device serial number global dictionary is updated, including: If log data with updated device number is received, the device number corresponding to the updated device number is mapped one by one to a numeric sequence number to obtain a set of numeric sequence numbers; Update the global dictionary of device serial numbers based on the set of numerical serial numbers.

6. The method according to claim 5, characterized in that, The step of updating the device serial number global dictionary based on the set of numerical serial numbers includes: Match the numerical serial numbers in the global dictionary of device serial numbers with the numerical serial numbers in the set of device serial numbers; Add any mismatched numeric serial numbers to the device serial number global dictionary.

7. A data deduplication device, characterized in that, Includes a unit for performing the method as described in any one of claims 1-6.

8. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-6.

Images