Data duplicate removal method, electronic equipment and storage medium

Abstract

The invention provides a data duplicate removal method, electronic equipment and a storage medium. According to the method, smooth data transition is achieved through a double-bloom filter alternating mechanism. In first time period processing, query and addition operations are only executed on a first bloom filter of a current time window. And second time period initialization: creating a new second bloom filter at the first unit time of the second time period as a next window preparation filter. According to the double-writing preheating mechanism, data are written into the first Bloom filter and the second Bloom filter at the same time in the second time period, and the missed judgment problem during window switching is avoided. According to window switching logic, old filter resources are released, the preliminary filter is marked as a new main filter, and monthly / periodic seamless connection is achieved. According to the method, the misjudgment rate is controlled through a double-bloom filter time window switching mechanism, the data duplicate removal accuracy and the system resource utilization rate can be improved, and the method is suitable for the efficient duplicate removal requirement in a mass data collection scene.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a data deduplication method, electronic equipment and storage medium. Background Art

[0002] As the volume of internet information continues to grow exponentially, web crawlers, as core tools for data collection, face significant challenges in terms of efficiency and resource utilization. Data deduplication, a key component of crawler systems, directly impacts data collection accuracy and system performance. Current mainstream deduplication technologies, based on their implementation principles, include relational database index deduplication, distributed key-value store deduplication, and traditional Bloom filter deduplication.

[0003] Relational database index deduplication solutions, based on relational databases like MySQL, verify data uniqueness by establishing unique indexes (such as URL hash indexes). When new data is stored, the index is used to quickly check for duplicate records. This solution has a low technical implementation barrier and is suitable for small-scale data scenarios, supporting complex queries and transaction management. However, in massive data scenarios, index size scales linearly with data volume, leading to a sharp increase in disk I / O overhead and query response times degrading from milliseconds to seconds. Storage costs are high, with the index space occupied by a single piece of data taking up up to two to three times the actual data. Furthermore, it struggles to scale to data volumes exceeding 100 million.

[0004] Distributed key-value store deduplication solutions utilize distributed systems such as HBase and Cassandra, using data fingerprints (such as MD5 hashes) as keys and storing them in distributed clusters. Data sharding is achieved through consistent hashing algorithms, supporting horizontal scalability. This solution can theoretically support petabyte-level data storage and offers automatic fault tolerance and load balancing, making it suitable for distributed crawler scenarios. However, the system architecture is complex, requiring solutions to issues such as data shard consistency and cross-node query latency. Operational and maintenance costs are high, requiring a dedicated team to maintain cluster topology and data migration strategies. For high-frequency queries, cross-node network overhead and cache invalidation remain issues, typically limiting the query per second (QPS) of a single machine to under 10,000.

[0005] Traditional Bloom filter deduplication schemes are based on a probabilistic data structure consisting of a bitmap and multiple hash functions. Using k hash functions, elements are mapped to k positions in a bit array and set to 1. During queries, the hash map is used to determine whether the element exists. This scheme is extremely space-efficient, requiring only a few MB of memory to store millions of data points. Furthermore, its query time complexity is O(k), which is close to constant time, making it suitable for high-frequency deduplication scenarios. However, it lacks the ability to proactively clean up expired data. As data continues to be written, the density of "1s" in the bitmap approaches 100%, and the false positive rate (false positives) increases exponentially. According to field measurements, when bitmap utilization exceeds 70%, the false positive rate can soar from an initial 0.1% to over 5%. Furthermore, this scheme lacks a robust backup and recovery mechanism. If a system failure causes bitmap data loss, the full data fingerprint must be rebuilt, potentially causing repeated crawler collection, wasting server resources and causing abnormal access to the target website.

[0006] Therefore, existing data deduplication solutions inherently face a triangular dilemma: storage efficiency, query performance, and system reliability. Relational databases and distributed key-value storage struggle to overcome the bottlenecks of storage costs and query efficiency, while traditional Bloom filters are limited by accumulated false positives and data reliability issues. In scenarios where billions of data are collected daily, existing technologies are unable to meet the crawler system's requirements for deduplication modules: low power consumption, high reliability, and self-optimization. Summary of the Invention

[0007] In view of the above technical problems, the technical solution adopted by the present invention is: According to a first aspect of the present invention, a data deduplication method is provided, the method comprising the following steps: S100, determine the position of the current monitoring moment in the time window: if it is located in the first time period of the current time window, execute S200; if it is located in the first unit time of the second time period of the current time window, execute S300; if it is located in the second time period of the current time window and not the first unit time, execute S400; if it is located in the first unit time of the next adjacent time window, execute S500; wherein, the first time period is the first M consecutive unit times of the current time window, the second time period is the last N consecutive unit times of the current time window, M and N are positive integers, and the total length of the current time window is M+N unit times.

[0008] S200, when receiving the data URL to be processed within the unit time of the current monitoring moment, perform a query operation on the first Bloom filter corresponding to the current time window based on the data URL to obtain the query result; if the query result indicates that the data URL needs to be added, add the data URL to the first Bloom filter and execute S100; the initial state of the first Bloom filter is a newly initialized Bloom filter.

[0009] S300: Create a newly initialized second Bloom filter as a preliminary Bloom filter for the next adjacent time window, and execute S400.

[0010] S400: When a data URL to be processed within the unit time of the current monitoring moment is received, a query operation is performed on the first Bloom filter corresponding to the current time window based on the data URL to obtain a query result. If the query result indicates that the data URL needs to be added, the data URL is added to the first Bloom filter and the second Bloom filter; otherwise, the data URL is only added to the second Bloom filter, and S100 is executed. S500 , releasing resources occupied by the first Bloom filter, marking the second Bloom filter as a new first Bloom filter for data processing in subsequent time windows, and executing S200 .

[0011] According to a second aspect of the present invention, an electronic device is provided, comprising a processor and a memory; the processor is configured to execute the steps of the method according to the first aspect of the present invention by calling a program or instruction stored in the memory.

[0012] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method according to the first aspect of the present invention.

[0013] The present invention has at least the following beneficial effects: A data deduplication method provided by an embodiment of the present invention can effectively solve the problem of data omission when the time window is switched by simultaneously writing the current window filter (first filter) and the next window filter (second filter) at the end of the time window, so that the filter can correctly identify duplicate data at the beginning of the month and avoid repeated collection caused by window switching.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a data deduplication method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0019] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. A process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0020] The present invention provides a data deduplication method, which aims to solve the problem that the traditional Bloom filter does not support the deletion of elements, resulting in a continuous increase in the misjudgment rate and how to achieve effective backup and rapid recovery of data.

[0021] Furthermore, if Figure 1 As shown, a data deduplication method provided by an embodiment of the present invention includes the following steps: S100, determine the position of the current monitoring moment in the time window: if it is located in the first time period of the current time window, execute S200; if it is located in the first unit time of the second time period of the current time window, execute S300; if it is located in the second time period of the current time window and not the first unit time, execute S400; if it is located in the first unit time of the next adjacent time window, execute S500; wherein, the first time period is the first M consecutive unit times of the current time window, the second time period is the last N consecutive unit times of the current time window, M and N are positive integers, and the total length of the current time window is M+N unit times.

[0022] In this embodiment of the present invention, the time window is divided based on the time period of a natural month, with each unit time lasting a full natural day (24 hours). The monitoring time is set as a fixed time point within each unit time. Specifically, the monitoring time is the end time of each natural day, 00:00:00 (i.e., the start time of the next day).

[0023] In an embodiment of the present invention, the method is used to perform deduplication processing on data URLs collected by a web crawler. It is understood that a data URL (Uniform Resource Locator) is an address identifier used to locate and access network resources.

[0024] In this embodiment of the present invention, N is positively correlated with the web crawler's collection cycle. Specifically, the longer the collection cycle, the larger the value of N; the shorter the collection cycle, the smaller the value of N. This correlation mechanism is designed to ensure accurate and efficient data deduplication. A web crawler's collection cycle refers to the time interval between two collections of the same target data. A long collection cycle means the crawler must process a relatively large amount of data in a single collection task, increasing the likelihood and complexity of data duplication. For example, if a crawler performs network-wide data collection monthly, near the end of the collection task, the accumulated data volume increases, leading to a high probability of overlap between new and old data. In this case, setting N to a larger value (e.g., 5-7 days) and enabling the double Bloom filter for a longer period before the end of the window can fully cover potential data duplication scenarios and effectively avoid missed detections due to window switching. Conversely, when the collection cycle is short, such as when the crawler collects data daily, the amount of data processed each time is relatively small, and the risk of data duplication is lower. In this case, N can be set to a smaller value (such as 1 to 2 days), which can not only meet the needs of smooth data transition, but also reduce the write burden of the double Bloom filter and reduce system resource consumption.

[0025] Furthermore, dynamically adjusting the N value can adapt to changes in web crawler collection strategies. When the crawler adjusts its collection frequency or scope due to business needs, the system can automatically or manually adjust the N value based on the new collection cycle, ensuring that the data deduplication mechanism is always operating optimally. For example, during a major e-commerce promotion, the crawler may temporarily increase its collection frequency to obtain real-time product information. Increasing the N value in this situation can better cope with the sudden surge in data volume and ensure accurate deduplication.

[0026] In an exemplary embodiment of the present invention, N can be determined based on an empirical formula derived from historical data statistics. For example, by analyzing historical crawler data, the relationship between data repetition rate and N values ​​at different collection periods can be statistically analyzed, and an empirical formula can be obtained by fitting. For example, the empirical formula can be: N = rounddown(c × T) + 1, where rounddown ( ) represents rounding down, c is a preset coefficient, and is the fitted value. In one exemplary embodiment, c = 0.2. T is the collection period, expressed in days.

[0027] In another exemplary embodiment of the present invention, N can be determined based on a computational model for estimating data volume. In one example, N = roundup(log(Q × R) / log²), where Q is the estimated data volume for each acquisition task and R is the historical data repetition rate. Roundup ( ) indicates rounding up.

[0028] In another exemplary embodiment of the present invention, N can be determined based on dynamic association rules between partitions. For example, different N value rules are formulated according to different ranges of the collection period: When the collection period T≤T1, set N=1. TI is the first collection period and can be set to 5 days.

[0029] When T1≤T<T2, set N=[T / 5]+1. T2 is the second collection period and can be set to 15 days. [ ] indicates rounding.

[0030] When T2≤T, set N=[T / 3]+1. T2 is the first collection period and can be set to 15 days.

[0031] For example, when T=3 days, N=1 day; T=10 days, N=[10 / 5]+1=3 days; T=20 days, N=[10 / 3]+1=7 days. This partition interval rule can be flexibly adjusted according to actual business characteristics, so as to facilitate the rapid determination of the N value in different acquisition cycle scenarios, while taking into account the accuracy of data deduplication and system resource consumption. S200, when the data URL to be processed within the unit time of the current monitoring moment is received, a query operation is performed on the first Bloom filter corresponding to the current time window based on the data URL to obtain the query result; if the query result indicates that the data URL needs to be added, the data URL is added to the first Bloom filter, and S100 is executed; the initial state of the first Bloom filter is a newly initialized Bloom filter.

[0032] S300: Create a newly initialized second Bloom filter as a preliminary Bloom filter for the next adjacent time window, and execute S400.

[0033] At step S400, upon receiving a data URL to be processed within the unit time of the current monitoring moment, a query operation is performed on the first Bloom filter corresponding to the current time window based on the data URL to obtain a query result. If the query result indicates that the data URL needs to be added, the data URL is added to the first Bloom filter and the second Bloom filter. Otherwise, that is, if the query result indicates that the data URL does not need to be added, the data URL is only added to the second Bloom filter, and step S100 is executed. In this way, the second Bloom filter will be pre-stored with some data from the end of the previous month.

[0034] S500 , releasing resources occupied by the first Bloom filter, marking the second Bloom filter as a new first Bloom filter for data processing in subsequent time windows, and executing S200 .

[0035] In the embodiment of the present invention, the number of Bloom filters currently in use is determined based on the time period of the current monitoring moment. When the current monitoring moment is in the last few days of the current time window, the second Bloom filter is started, that is, the Bloom filter double-write and monthly cycle mechanism is adopted. The core advantages include: Accurate duplicate removal guarantee: The double-write mechanism writes the current and next time window filters at the end of the time window to avoid missed judgments / misjudgments due to window switching, which is especially suitable for the data peak scenario at the end of the month to ensure the accuracy of repeated identification during data handover; Data continuity management: The monthly cycle mechanism enables seamless connection of filters and complete retention of historical data , providing a continuous data basis for trend analysis, prediction, etc., avoiding processing interruptions; load balancing and stability: dual writing shares the writing pressure of a single filter, reducing the risk of performance degradation under large data volumes; the monthly pre-initialization mechanism can adjust parameters in advance according to data estimates to cope with periodic traffic fluctuations; efficient resource utilization: dual writing is only enabled at the end of the window, and a single filter runs most of the time, reducing memory and computing overhead; dynamic configuration of filter parameters on a monthly basis to avoid resource waste; maintainability optimization: the periodic mechanism facilitates data archiving, cleaning and system monitoring. After the monthly processing is completed, targeted filter management and status checks can be performed to improve operation and maintenance efficiency.

[0036] Furthermore, in an embodiment of the present invention, the query operation and the add operation are implemented based on the Redis cluster, the first Bloom filter and the second Bloom filter have the same structure, each Bloom filter is composed of multiple independent data blocks, the data blocks of each Bloom filter are divided according to a preset hash range, and each data block is evenly mapped to different master nodes of the Redis cluster through a consistent hashing algorithm. The data blocks on each master node are stored in a BitMap structure to record the existence status of the data URL.

[0037] In specific implementation, the method of the present invention can be implemented by deploying a Redis cluster consisting of 3 servers, each server is configured as: 16 cores, 32G memory, and 500GB storage. The Redis cluster is configured as a 3-master-3-slave architecture, that is, 3 master nodes, each master node is connected to a slave node. Each master node is associated with multiple hash slots, each hash slot is associated with multiple hash bits, and each master node is connected to a slave node. The Redis cluster is deployed in Redis Cluster mode, and 3 sentinel nodes are configured to monitor the cluster status. The Redis version is selected to be 7.0 or above, and the AOF persistence mechanism is enabled to ensure data security.

[0038] In an embodiment of the present invention, the bit array size m and the number of hash functions k of each Bloom filter satisfy the following conditions: m=-n×ln(p) / (ln(2)²), k=m / n×ln(2); where n is the estimated data volume and p is the target error rate.

[0039] In a specific embodiment, the Bloom filter is configured as follows: the estimated monthly new data volume is 300 million URLs, the Bloom filter hash times are 16 times, the false positive rate is set to 0.001%, and the initial size of each monthly filter is approximately 3.2 GB of memory space.

[0040] Those skilled in the art should understand that the specific partitioning strategy for Bloom filter data blocks (such as by hash range, data popularity, etc.) and the mapping mechanism for data blocks to Redis cluster master nodes (such as the consistent hashing algorithm) are all existing technologies that have been widely used in this field and will not be elaborated here.

[0041] Furthermore, based on the received data URL, performing a query operation on the first Bloom filter corresponding to the current time window to obtain a corresponding query result specifically includes: S10, using a preset hash function and k different seed values, performing hash calculation on the data URL to obtain k different hashes.

[0042] In an embodiment of the present invention, the preset hash function is the MurmurHash3 function. The k seed values ​​can range from 0 to k-1. When calculating the hash value of a data URL, each calculation is performed using the MurmurHash3 function and a different seed value. For example, if k=16, mmh3.hash(url,0), mmh3.hash(url,1), ..., mmh3.hash(url,15) are calculated to generate 16 hash values. Compared to implementing k different hash functions, this method has lower computational overhead while ensuring uniform distribution of hash values.

[0043] S11, based on the GETBIT instruction provided by the Redis cluster, map the k hash values ​​to k positions of the bit array corresponding to the first Bloom filter respectively, and obtain the state values ​​of the k first mapping positions.

[0044] S11 may specifically include: S1101, perform a modulo operation on each of the k hash values ​​to ensure that each hash value is mapped to the valid range of the Bloom filter bit array, and obtain k valid indexes. Wherein, valid index = hash value % bit array length. % represents a modulo operation. Wherein, the bit array length is the total number of bits in the Bloom filter (e.g., 2 24 ) S1102: Determine the data block ID to which each valid index belongs, where data block ID = valid index / / data block size. The data block size is a preset value, and / / represents integer division.

[0045] S1103: Determine the Redis master node ID corresponding to each data block ID using a consistent hashing algorithm. Specifically, a hash ring algorithm can be used to map the data block ID to a ring node space using a hash function (such as MurmurHash), where the node ID is equal to the nearest active master node in the clockwise direction.

[0046] S1104: For k valid indexes, execute the GETBIT command in batches through the Redis cluster pipeline. The command format is: GETBIT bloomfilter:window:{window ID}:chunk:{data chunk ID} {valid index}. {window ID} is the unique identifier of the current time window (e.g., 202506), and {data chunk ID} is the chunk number calculated in S1102.

[0047] S1105, parse the return value (0 or 1) of the GETBIT instruction and convert it into a Boolean array, where 0 indicates that the position is not marked and the corresponding element definitely does not exist, and 1 indicates that the position is marked and the corresponding element may exist (there is a possibility of misjudgment).

[0048] S1106: Assemble the k state values ​​into a Boolean array of length k in hash order, which will serve as a basis for subsequent existence judgment.

[0049] S12, based on the status values ​​of the k first mapping positions, determine whether the data URL already exists in the first Bloom filter: if all status values ​​are 1, a query result is generated indicating that the data URL may already exist and does not need to be added; if there is at least one status value of 0, a query result is generated indicating that the data URL does not exist and needs to be added.

[0050] Furthermore, the adding operation is specifically implemented by setting the values ​​of k hash positions corresponding to the data URL to 1 through the SETBIT instruction provided by the Redis cluster, specifically including: S1, using the same k hash functions and seed values ​​as the query operation, calculate k hash values ​​of the URL of the data to be added.

[0051] S2, convert the k hash values ​​into valid indexes in the bit array through modulo operation.

[0052] S3, determine the data block ID according to the valid index.

[0053] S4, determine the Redis master node ID corresponding to each data block ID through the consistent hashing algorithm.

[0054] S5 maintains a buffer of URLs to be added and triggers batch processing according to the following rules: When the buffer accumulates more than 1000 URLs or the waiting time reaches 50ms, group and merge SETBIT operations by (master node ID, data chunk ID). Example grouping: {node1:{chunk1:[offset1,offset2],chunk2:[offset3]},node2:{chunk3:[offset4]}}.

[0055] S6, through the Pipeline mechanism of the Redis cluster, batch performs the following operations: SETBITbloomfilter:window:{window ID}:chunk:{data block ID}{valid index}1.

[0056] S6, through the Redis cluster's Pipeline mechanism, performs batch SETBIT operations by group: for each master node, create an independent Pipeline connection; execute the instruction format: SETBITbloomfilter:window:{window ID}:chunk:{data block ID}{valid index}1.

[0057] S7, for k SETBIT operations on the same data block, atomicity is ensured through Redis transactions (MULTI / EXEC).

[0058] In summary, the data deduplication method provided by the embodiment of the present invention has at least the following advantages: (1) By utilizing the high performance characteristics of the Redis memory database, it is possible to achieve millisecond-level query response time based on the GETBIT instruction, which can meet the high concurrency requirements of crawlers.

[0059] (2) Through the master-slave replication and sentinel mechanism of the Redis cluster, the system achieves high availability, ensuring 99.99% service availability and effectively avoiding the risk of single point of failure. When the master node fails, the sentinel mechanism can automatically promote the slave node to the master node. The entire switching process is transparent to the business, and the data loss is less than or equal to 1 second. In addition, by configuring the Redis AOF persistence mechanism, even in extreme cases, the system can recover data from the disk, ensuring the integrity and recoverability of the deduplication data.

[0060] (3) By adopting a double-write Bloom filter and a monthly cycle mechanism, this service effectively controls and maintains an extremely low false positive rate. By pre-creating filters for the next month at the end of the month, we accumulate basic data while also reducing historical invalid data, keeping the false positive rate within a preset 0.001%. This means that when processing 1 billion pieces of data, the number of false positives does not exceed 10,000. This is a completely acceptable level for massive data deduplication scenarios and significantly better than the false positive rate accumulation problem of traditional single Bloom filters after long-term operation.

[0061] (4) In actual application scenarios, Redis nodes can be dynamically added or subtracted based on business needs, theoretically supporting an unlimited amount of data deduplication needs. Whether the amount of data increases or the number of concurrent requests increases, simply expanding the Redis cluster is sufficient, without requiring large-scale modifications to the application layer, greatly reducing the complexity of system expansion.

[0062] (5) Compared with the complex solutions in the Hadoop ecosystem, the deployment and maintenance costs of Redis-based solutions are low.

[0063] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0064] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0065] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0066] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data deduplication method, characterized in that: The method comprises the following steps: S100, determine the position of the current monitoring moment in the time window: if it is in the first time period of the current time window, execute S200; if it is in the first unit time of the second time period of the current time window, execute S300; if it is in the second time period of the current time window and not the first unit time, execute S400; if it is in the first unit time of the next adjacent time window, execute S500; wherein, the first time period is the first M consecutive unit times of the current time window, the second time period is the last N consecutive unit times of the current time window, M and N are positive integers, and the total length of the current time window is M+N unit times; S200: When a data URL to be processed within the unit time of the current monitoring moment is received, a query operation is performed on the first Bloom filter corresponding to the current time window based on the data URL to obtain a query result; if the query result indicates that the data URL needs to be added, the data URL is added to the first Bloom filter and S100 is executed; the initial state of the first Bloom filter is a newly initialized Bloom filter; S300: Create a newly initialized second Bloom filter as a preliminary Bloom filter for the next adjacent time window, and execute S400; S400: When a data URL to be processed within the unit time of the current monitoring moment is received, a query operation is performed on the first Bloom filter corresponding to the current time window based on the data URL to obtain a query result. If the query result indicates that the data URL needs to be added, the data URL is added to the first Bloom filter and the second Bloom filter; otherwise, the data URL is only added to the second Bloom filter, and S100 is executed. S500 , releasing resources occupied by the first Bloom filter, marking the second Bloom filter as a new first Bloom filter for data processing in subsequent time windows, and executing S200 .

2. The method according to claim 1, characterized in that The query operation and the add operation are implemented based on the Redis cluster. The first Bloom filter and the second Bloom filter have the same structure. Each Bloom filter is composed of multiple independent data blocks. The data blocks of each Bloom filter are divided according to a preset hash range. Each data block is evenly mapped to different master nodes of the Redis cluster through a consistent hashing algorithm. The data blocks on each master node are stored in a BitMap structure to record the existence status of the data URL.

3. The method according to claim 2, characterized in that The query operation is performed on the first Bloom filter corresponding to the current time window based on the received data URL to obtain the corresponding query result, specifically including: S10, using a preset hash function and k different seed values, performing hash calculation on the data URL to obtain k different hash values, where k is the number of hash functions of the Bloom filter; S11, based on the GETBIT instruction provided by the Redis cluster, mapping the k hash values ​​to k positions of the bit array corresponding to the first Bloom filter, and obtaining the state values ​​of the k first mapping positions; S12, based on the status values ​​of the k first mapping positions, determine whether the data URL already exists in the first Bloom filter: if all status values ​​are 1, a query result is generated indicating that the data URL may already exist and does not need to be added; if there is at least one status value of 0, a query result is generated indicating that the data URL does not exist and needs to be added.

4. The method according to claim 2, characterized in that The adding operation is specifically implemented by setting the values ​​of k hash positions corresponding to the data URL to 1 through the SETBIT instruction provided by the Redis cluster, where k is the number of hash functions of the Bloom filter.

5. The method according to claim 3, characterized in that The preset hash function is the MurmurHash3 function.

6. The method according to claim 1, characterized in that The method is used to perform deduplication processing on data URLs collected by web crawlers.

7. The method according to claim 1, characterized in that The time window is divided based on the time period of a natural month, and the duration of each unit time is a complete natural day.

8. The method according to claim 6, characterized in that The N is positively correlated with the collection period of the web crawler.

9. An electronic device, characterized in that: including processor and memory; The processor is configured to execute the steps of the method according to any one of claims 1 to 8 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 8.

Images