Hbase-based mass call record distributed deduplication method and system

Abstract

This invention discloses a distributed deduplication method and system for massive call detail records (CDRs) based on HBase, involving distributed massive data deduplication technology. Addressing the performance and real-time deficiencies of existing deduplication techniques operating in distributed systems, this solution proposes a method that submits batch CDR GET requests to HBase from the deduplication table. CDRs that return results are identified as duplicates. For CDRs that do not return results, batch PUT requests are submitted to HBase for the corresponding deduplication keys. Then, batch GET requests are submitted to HBase again from the deduplication table to retrieve all historical version records and compare them with the earliest version entered into the database. If it is not the earliest version, the CDR is identified as a duplicate. A three-stage check method of batch read, write, and read is used to improve throughput. This method can meet the growing business demands of telecommunications, handle massive real-time CDR deduplication, ensure that the deduplication module does not become a bottleneck for the entire system, and reserve processing capacity for prepaid services.

Description

Technical Field

[0001] This invention relates to distributed massive data deduplication technology, and more particularly to a distributed deduplication method and system for massive call detail records based on HBase. Background Technology

[0002] In communication billing systems, duplicate call detail records (CDRs) collected from various network elements often appear due to various uncontrollable reasons. Therefore, an important module of the billing system is to remove duplicate CDRs to ensure the accuracy of billing.

[0003] In existing technologies, the following strategies are typically used for deduplication:

[0004] 1. Memory deduplication: The deduplication key of all call detail records is written to local memory for comparison. Although it is efficient, it can only run in single-machine mode and cannot meet the needs of distributed applications with massive call detail records.

[0005] 2. Database deduplication: The deduplication key for all call detail records is written to a relational database. Although this can satisfy cluster access, traditional relational databases cannot meet performance requirements when faced with massive read, write and locking requests.

[0006] 3. Deduplication using HBase's checkAndPut method: While the atomicity of HBase's table-level checkAndPut method can handle massive amounts of data, its real-time performance still falls short of commercial settlement requirements. Summary of the Invention

[0007] The purpose of this invention is to provide a distributed deduplication method and system for massive call detail records based on HBase, so as to solve the problems existing in the prior art.

[0008] The distributed deduplication method for massive call detail records based on HBase described in this invention includes the following steps:

[0009] The steps for generating the deduplication table are as follows: obtain call detail records (CDRs) from each distributed network point and form batch CDRs; calculate the deduplication key for each CDR and generate the deduplication table corresponding to the batch of CDRs.

[0010] The three-stage deduplication process is as follows: First, submit a batch GET request for the deduplication table to HBase. Call records that return results are considered duplicates. Second, submit a batch PUT request for the deduplication key corresponding to the call records that do not return results to HBase. Third, submit another batch GET request for the deduplication table to HBase to retrieve all historical version records and compare the call records with the earliest version entered into the database. If it is not the earliest version entered into the database, the call record is considered a duplicate.

[0011] Deduplication step: Remove duplicate call records based on the judgment results of the three-stage deduplication step.

[0012] Before submitting a batch call detail record (CDR) GET request to HBase again, remove the CDRs that have been identified as duplicates from the deduplication table.

[0013] When HBase receives a batch put request for deduplication keys, it simultaneously records the timestamp of each deduplication key being entered into the database. When retrieving all historical version records to compare whether the call detail record (CDR) is the earliest version entered into the database, it also compares the deduplication key and its corresponding timestamp.

[0014] When recording the deduplication key and its timestamp, a sequence is appended after the timestamp.

[0015] Using HBase's coprocessor mechanism, an observer is deployed on the regionserver; the observer is used to intercept batch put requests for deduplicated keys; after interception, the timestamps are then concatenated with the sequence.

[0016] The present invention discloses a distributed deduplication system for massive call detail records (CDRs) based on HBase, which completes the deduplication process using the steps of the distributed deduplication method for massive CDRs.

[0017] The distributed deduplication method and system for massive call detail records (CDRs) based on HBase described in this invention has the advantage of using a three-stage check method of batch read, write, and read to improve throughput. It can meet the ever-increasing business demands of telecommunications, effectively deduplicating massive real-time CDRs, ensuring that the deduplication module does not become a bottleneck for the entire system, and reserving processing capacity for prepaid services. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a distributed deduplication method for massive call detail records based on HBase, as described in this invention. Detailed Implementation

[0019] like Figure 1 As shown, the distributed deduplication method for massive call detail records based on HBase described in this invention improves throughput through a three-stage check method of batch reading, writing, and reading, so that deduplication is no longer a technical bottleneck in the processing of massive call detail records.

[0020] First, call detail records (CDRs) are retrieved from various distributed network points to form batch CDRs. Then, a deduplication key is calculated for each CDR, generating a deduplication table corresponding to that batch of CDRs. A deduplication key is a unique identifier corresponding to a single CDR, and it can be obtained using various existing specific coding algorithms, such as hash values. A deduplication table contains a massive number of CDR deduplication keys.

[0021] Then, a three-stage deduplication process is used to quickly and accurately retrieve duplicate call detail records (CDRs). The read-write-read three-stage process is the core of this invention, which can meet the performance requirements for real-time deduplication of massive amounts of data.

[0022] The first step – Reading: Submit a batch call detail record (CDR) GET request to HBase using the deduplication table. CDRs that return a result are considered duplicates because the corresponding deduplication key already exists in HBase. After removing duplicates from the deduplication table, the deduplication table is then applied to the next step.

[0023] The second part - write: If no result is returned in the first part, submit a batch put request to HBase for the corresponding deduplication key;

[0024] The third step – Reading: Submit a GET request to the batch call detail records (CDRs) in the second step to retrieve all historical version records. Compare the unique CDR identifier and the timestamp of entry into the database to determine if this CDR is the earliest CDR entered into the database; if not, it is considered a duplicate.

[0025] Finally, duplicate call records are removed based on the results of the three-stage deduplication process.

[0026] This method achieves the goal of high system performance, but it has a flaw in engineering: because HBase's insertion timestamp precision is only down to the millisecond level, if multiple deduplication keys are the same, only one record can be read if inserted in the same millisecond. Therefore, an auxiliary patch is needed: utilizing HBase's coprocessor mechanism, an observer is deployed on the regionserver to intercept put operations on the deduplication table. A 4-digit sequence number is appended to the timestamp of each record to ensure the uniqueness of each record's insertion timestamp.

[0027] The present invention discloses a distributed deduplication system for massive call detail records (CDRs) based on HBase, which completes the deduplication process using the steps of the distributed deduplication method for massive CDRs.

[0028] The advantages and effects of this invention compared to the prior art.

[0029] 1. It meets the performance requirements for real-time deduplication of massive amounts of data (performance meets standards);

[0030] 2. It fully utilizes the powerful distributed open-source component HBase as the basic storage layer;

[0031] 3. Minimal development and deployment workload: Only a relatively simple batch deduplication algorithm needs to be implemented, and a custom coprocessor needs to be deployed on HBase.

[0032] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A distributed deduplication method for massive call detail records based on HBase, characterized in that, Includes the following steps: The steps for generating the deduplication table are as follows: obtain call detail records (CDRs) from each distributed network point and assemble them into batch CDRs; Calculate the deduplication key for each call detail record (CDR) and generate a deduplication table corresponding to the batch of CDRs. Three-stage plagiarism detection steps: Submit a batch call detail record (CDR) GET request to HBase to the deduplication table. Call detail records that return a result are identified as duplicates. Call detail records identified as duplicates are removed from the deduplication table. If no results are returned, submit a batch put request to HBase for the corresponding deduplication key; when HBase receives the batch put request for the deduplication key, it will record the timestamp of the deduplication key being entered into the database. Using HBase's coprocessor mechanism, an observer is deployed on the regionserver; the observer is used to intercept the batch put requests for deduplicated keys; after interception, a sequence is appended to the timestamp of each record to ensure the uniqueness of the timestamp of each record entering the database; Submit a batch call detail record (CDR) GET request to HBase again to retrieve all historical version records; compare the deduplication key and its corresponding timestamp to determine if this CDR is the earliest version entered into the database. If it is not the earliest version entered into the database, then the CDR is determined to be a duplicate. Deduplication step: Remove duplicate call records based on the judgment results of the three-stage deduplication step.

2. A distributed deduplication system for massive call detail records based on HBase, characterized in that, Use the following steps to complete the call detail record (CDR) deduplication process: The steps for generating the deduplication table are as follows: obtain call detail records (CDRs) from each distributed network point and form batch CDRs; calculate the deduplication key for each CDR and generate the deduplication table corresponding to the batch of CDRs. Three-stage plagiarism detection steps: Submit a batch call detail record (CDR) GET request to HBase to the deduplication table. Call detail records that return a result are identified as duplicates. Call detail records identified as duplicates are removed from the deduplication table. If no results are returned, submit a batch put request to HBase for the corresponding deduplication key; when HBase receives the batch put request for the deduplication key, it will record the timestamp of the deduplication key being entered into the database. Using HBase's coprocessor mechanism, an observer is deployed on the regionserver; the observer is used to intercept the batch put requests for deduplicated keys; after interception, a sequence is appended to the timestamp of each record to ensure the uniqueness of the timestamp of each record entering the database; Submit a batch call detail record (CDR) GET request to HBase again to retrieve all historical version records; compare the deduplication key and its corresponding timestamp to determine if this CDR is the earliest version entered into the database. If it is not the earliest version entered into the database, then the CDR is determined to be a duplicate. Deduplication step: Remove duplicate call records based on the judgment results of the three-stage deduplication step.

Images