High-efficiency charging ticket duplicate removal method
By adopting a two-level routing strategy and a combination of distributed Redis cluster, Bloom filter and TeleDB cluster in the call-line diversion system, the problems of low efficiency and insufficient availability of massive call-line diversion checks are solved, and efficient and scalable call-line diversion checks are achieved.
Patent Information
- Application Number
- CN202411959787.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-11
AI Technical Summary
When handling massive calls, it is difficult to efficiently determine the call order, resulting in repeated billing and customer complaints, and there are shortcomings in high availability disaster recovery.
A two-level routing strategy is adopted, combining distributed Redis cluster, Bloom filter and TeleDB cluster to achieve efficient diversion check of telephone orders. The specific steps include the client extracting the key fields of the call order, determining the target server through a two-level routing policy, and using the Redis cluster, Bloom filter and TeleDB cluster to check the diversion.
It realizes efficient plagiarism checking of massive calls, supports linear expansion of scale, and has the ability to switch in seconds during high-availability disaster recovery, avoiding repeated billing and customer complaints.
Smart Images
Figure CN119814641A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of call bill duplicate checking, and in particular relates to a highly efficient method for deduplicating bills. Background Art
[0002] The call bill refers to the original record information of the communication, which is the basis for the billing of the user's call, so the call bill needs to be collected. However, repeated collection during the call bill collection process or abnormal operation of the switch's fault light may generate duplicate call bills. Therefore, it is necessary to check the call bills for duplicates to avoid duplicate billing due to duplicate call bills, causing customer complaints and loss of customer sources.
[0003] Most existing call record duplication checking technologies are based on databases or files. In the database method, whether it is a traditional database (Oracle, MySQL) or a No-SQL database, a large number of database tables need to be established to store key-value data. Disk IO has high latency and cannot switch in real time in terms of high availability and disaster recovery. The file duplication checking method uses the unique index of the table to check for duplication, which requires a large number of files to index, requires huge investment in host resources, and is also weak in terms of high availability.
[0004] The existing call record duplication checking method has a high duplication checking efficiency for a small number of call records. However, with the rapid development of telecommunications services, the amount of business data is becoming increasingly large, that is, the number of call records is increasing. How to efficiently determine duplicate call records from massive call record information is an urgent problem that needs to be solved. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a highly efficient method for deduplicating billing calls in view of the deficiencies of the above-mentioned prior art. Under the premise of maintaining efficient duplication detection, it can be linearly expanded to a larger data volume, thus solving the previous single-point risk of memory + file.
[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0007] A highly efficient method for re-ordering bills, the method comprising the following steps:
[0008] Step 1: The client extracts multiple key fields from the target call record and concatenates them into target identification information that can uniquely identify the target call record;
[0009] Step 2: Based on the two-level routing strategy, determine the target server cluster from multiple server clusters according to the city, determine the target server from the target server cluster according to the account, and then send the target identification information to the target server;
[0010] Step 3: After receiving the target identification information, the target server performs a duplicate check based on the Redis cluster, Bloom filter, and TeleDB cluster to determine whether the target call record is a duplicate.
[0011] To optimize the above technical solutions, the specific measures taken also include:
[0012] The key fields described in step 1 above are selected according to the business type, and at least include the city and account information.
[0013] The above key fields are managed in the form of parameter tables.
[0014] The above two-level routing strategy determines the target server from the target server cluster according to the account as follows:
[0015] SERV_ID=MOD(MOD(USER_ID,N1),N2)
[0016] Among them, SERV_ID is the target server ID;
[0017] MOD means modulus;
[0018] USER_ID is the account ID;
[0019] N1 is lg(account ID) / 2;
[0020] N2 is a prime number greater than 1000 and less than 10000.
[0021] The above-mentioned Redis cluster is used as a distributed cache, in which a binary array BitMap is preset. The BitMap includes m binary numbers. In the initial state, the m binary numbers included in the BitMap are all 0, which is implemented by BitSet.
[0022] The above Bloom filter uses a hash function method to map an element to a point on an array of length m. When this point is 1, the element is in the set, otherwise it is not in the set.
[0023] The specific process of the above hash function method is as follows:
[0024] Create a continuous BF memory to store all the information of the set elements. At the beginning, there are no elements in the set, that is, all bits are "0";
[0025] When a new element comes, before determining whether it is in the set, the element is hashed to generate multiple hash values;
[0026] The hash value is modulo the BF memory size, and the modulo value is used to obtain the bit information from the corresponding bit of the BF memory. If the corresponding bit is "0", the element is not in the set, and an insertion operation is performed to set the corresponding bit to "1"; if one of the corresponding bits is not "0", it is considered that the element is not in the set; otherwise, if the corresponding bits after the element is hashed are all "1", it is considered that the element may be in the set.
[0027] After receiving the target identification information, the above-mentioned target server performs HashMap calculation based on the Bloom filter to generate k binary numbers, and at the same time takes out k positions in the BitSet to make a judgment:
[0028] If there is a scenario where one bit is 0, it is judged as a non-duplicate order. Before returning to the client, all non-0 elements in the k positions need to be updated to 1. Then the target identification information requested by the client is synchronously written to the corresponding TeleDB cluster in an asynchronous manner;
[0029] If there is a scenario where the bit values of all k bits are 1, it is considered that there may be a duplicate order. After returning to the client, the client uses the keyword to search the corresponding business table in the TeleDB cluster. If the search is successful, it is considered not a duplicate order. If the search fails, it is considered a duplicate order and the corresponding record is written to the TeleDB cluster.
[0030] The present invention has the following beneficial effects:
[0031] The present invention proposes a routing strategy suitable for duplicate call record checking, adopts a distributed architecture, and utilizes a combination of distributed Redis clusters, Bloom Filter algorithms, and TeleDB clusters to ensure support for duplicate call record checking on a large scale, high efficiency of duplicate call record checking, linear expansion of the scale of duplicate call record checking, and the ability to switch duplicate call record checking services in seconds during high-availability disaster recovery switching. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A flow chart of a method for checking duplicate calls applied to a terminal provided in an embodiment of the present application;
[0033] Figure 2 A schematic diagram of the process of determining the target server cluster and the target server;
[0034] Figure 3 A schematic diagram of the position of the target identification information in the target memory data block indicated by the position indication information;
[0035] Figure 4 It is the overall business process diagram;
[0036] Figure 5 This is the flow chart of the Bloom filter algorithm. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0038] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.
[0039] like Figure 1-Figure 4 As shown, a highly efficient billing call record re-ordering method of the present invention comprises the following steps:
[0040] Step 1: The client extracts multiple key fields from the target call record and concatenates them into target identification information that can uniquely identify the target call record;
[0041] Step 2: Based on the two-level routing strategy, determine the target server cluster from multiple server clusters according to the city, determine the target server from the target server cluster according to the account, and then send the target identification information to the target server;
[0042] Step 3: After receiving the target identification information, the target server performs a duplicate check based on the Redis cluster, Bloom filter, and TeleDB cluster to determine whether the target call record is a duplicate.
[0043] In the embodiment, step 1 extracts multiple key fields from the target call record and concatenates the multiple key fields in the call record into information that can uniquely identify the target call record as target identification information;
[0044] Step 2: Determine the target server cluster from multiple server clusters based on key fields such as cities; determine the target server from the target server cluster based on key fields such as accounts;
[0045] The solution development adopts cluster deployment architecture, that is, the cluster structure composed of Client / Server (client / server), and adopts a two-level routing method.
[0046] [1] The routing is based on the city (telephone area code) corresponding to the user ID in the call record element.
[0047] [2] The user number is then modulo two prime numbers, and finally modulo the number of duplicate checking services planned for the corresponding telephone area code. This can effectively avoid the problem of uneven load caused by the user's special encoding method.
[0048] Secondary hash routing algorithm: SERV_ID = MOD(MOD(USER_ID, N1), N2)
[0049] Among them, SERV_ID: belongs to the server ID.
[0050] USER_ID: It is a key element taken from the call record structure, which is the user number modulo 2.
[0051] N1:lg(user ID) / 2, which is a fixed prime number after the exponential degradation operation of the user ID code. The general account number is 12 digits, so it is generally a prime number greater than 1000 and less than 10000.
[0052] N2: Same as N1, generally a prime number greater than 1000 and less than 10000.
[0053] After the above two-level hashing, the call slips of the same account can always be sent to the same duplicate checking service instance in the same cluster. Based on the amount of data, multi-level routing strategies are mapped to the backend duplicate checking server cluster, so that the capabilities of each duplicate checking service can be more effectively utilized.
[0054] The specific description of a single call record is as follows:
[0055] Client-side duplicate elimination requires selecting appropriate fields based on service types (such as traffic, voice, fixed network, SMS, value-added services, etc.) and managing them in the form of parameter tables. Multiple fields are extracted from user call records as feature codes, and they are combined in a fixed order to form duplicate detection keywords.
[0056] In the embodiment, step 3 sends the target identification information to the target server, so that the target server determines whether the target call record is a duplicate order based on the target identification information.
[0057] The server performs duplicate checking based on the redis cluster, bloom filter algorithm, and TeleDB data.
[0058] The Bloom Filter algorithm is described as follows:
[0059] If you need to determine whether an element is in a set, the usual approach is to save all elements and then compare them to see if they are in the set. As the number of elements in a set increases, the space and time required increase linearly, and the search speed also becomes slower and slower; Bloom filter can effectively solve the space and efficiency problems.
[0060] Bloom Filter is a probabilistic data structure used to efficiently detect whether an element belongs to a set, even if the actual data of the set may be very large. Specifically, Bloom filter is a binary vector data structure proposed by Howard Bloom in 1970. It has good space and time efficiency and is used to detect whether an element is a member of a set. This detection will only misjudge the data in the set, but not the data that is not in the set. In this way, each detection request returns two situations: "in the set (possibly wrong)" and "not in the set (definitely not in the set)". It can be seen that Bloom filter sacrifices accuracy in exchange for time and space.
[0061] Bloom filter uses a hash function method to map an element to a point on an array of length m. When this point is 1, the element is in the set (possibly wrong), otherwise it is not in the set (definitely not in the set).
[0062] The algorithm flow is:
[0063] (1) Create a continuous memory (BF memory) to store all the information of the set elements. At the beginning, there are no elements in the set, that is, all bits are "0", such as Figure 5 As shown in (a).
[0064] (2) When a new element comes, before determining whether it is in the set, the element is hashed. This process generates multiple hash values.
[0065] (3) The hash value in (2) above is modulo the size of the BF memory, and the modulo value is used to obtain the bit information of the corresponding bit in the BF memory. If the corresponding bit is "0", the element is not in the set. For example: after the element a is obtained by two hash functions (Hash1(a) = 0; Hash2(a) = 4), it is judged with the BF memory, such as Figure 5 as shown in (b).
[0066] (4) If the element is not in the set, you can insert it. You only need to set the corresponding position to "1", such as Figure 5 as shown in (c).
[0067] (5) When judging whether an element is in a set, if any bit is not "0", the element is considered not in the set; otherwise, if all corresponding positions after the element is hashed are "1", the element is considered to be in the set. Figure 5 as shown in (d).
[0068] In this embodiment, Redis is used as a distributed cache. The binary array preset in Redis is recorded as BitMap. BitMap includes m binary numbers. In the initial state, the m binary numbers included in BitMap are all 0.
[0069] Considering that traffic call records currently account for the largest proportion, about 1 billion records are generated every day. If the old memory + file deduplication solution is adopted, and the duplicate keywords are settled based on 64 bits, then each city will need to use 1 billion * 64 bytes = 59.6G of memory every day. At the same time, there is a single point access problem, and cluster deployment and high availability solutions are not supported.
[0070] Using the solution in this embodiment, when the number of samples n is 1 billion and the error rate p is set to 0.01%, according to the formula It can be deduced that m = 19.19n, rounded up to 20n. Among them, p, m, and n are the false positive rate, the length of the Bloom filter, and the number of elements, respectively. Therefore, 20 billion bits of memory are consumed, which is about 2.5G.
[0071] The number of hash functions k used can be calculated by the following formula:
[0072]
[0073] Therefore, k=14, which means that 14 hash functions are needed.
[0074] Specifically, when the server receives the call order elements extracted by the client, it performs HashMap calculation based on the Bloom filter to generate k binary numbers, and at the same time takes out k positions in the BitSet to make a judgment:
[0075] If there is a scenario where one bit is 0, it is judged as a non-duplicate order. Before returning to the client, it is necessary to update all non-zero elements in the k positions to 1. Next, the duplicate search keyword requested by the client is synchronously written into the corresponding TeleDB in an asynchronous manner.
[0076] If there is a scenario where the bit values of all k bits are 1, it is considered that there may be a duplicate order. After returning it to the client, the client uses the keyword to search the corresponding business table in TeleDB. If the search is successful, it is considered not a duplicate order. If it fails, it is considered a duplicate order and the record is written to TeleDB.
[0077] TeleDB is an easy-to-use and easy-to-install database cluster system based on the open source MySQL5.7 database system and the independently developed related functional component architecture.
[0078] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0079] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A highly efficient billing call record re-ordering method, characterized in that: The method comprises the following steps: Step 1: The client extracts multiple key fields from the target call record and concatenates them into target identification information that can uniquely identify the target call record; Step 2: Based on the two-level routing strategy, determine the target server cluster from multiple server clusters according to the city, determine the target server from the target server cluster according to the account, and then send the target identification information to the target server; Step 3: After receiving the target identification information, the target server performs a duplicate check based on the Redis cluster, Bloom filter, and TeleDB cluster to determine whether the target call record is a duplicate.
2. A highly efficient billing call record re-ordering method according to claim 1, characterized in that: The key fields described in step 1 are selected according to the business type, and include at least the city and account information.
3. The highly efficient billing call record re-ordering method according to claim 1, characterized in that: The key fields are managed in the form of parameter tables.
4. The highly efficient billing call record re-ordering method according to claim 1, characterized in that: The two-level routing strategy determines the target server from the target server cluster according to the account in the following way: SERV_ID=MOD(MOD(USER_ID,N1),N2) Among them, SERV_ID is the target server ID; MOD means modulus; USER_ID is the account ID; N1 is lg(account ID) / 2; N2 is a prime number greater than 1000 and less than 10000.
5. The highly efficient billing call record re-ordering method according to claim 1, characterized in that: The Redis cluster is used as a distributed cache, in which a binary array BitMap is preset, and the BitMap includes m binary numbers. In the initial state, the m binary numbers included in the BitMap are all 0, which is implemented by BitSet.
6. A highly efficient billing call record re-ordering method according to claim 5, characterized in that: The Bloom filter uses a hash function method to map an element to a point on an array of length m. When this point is 1, the element is in the set, otherwise it is not in the set.
7. A highly efficient billing call record re-ordering method according to claim 6, characterized in that: The specific process of the hash function method is as follows: Create a continuous BF memory to store all the information of the set elements. At the beginning, there are no elements in the set, that is, all bits are "0"; When a new element comes, before determining whether it is in the set, the element is hashed to generate multiple hash values; The hash value is modulo the BF memory size, and the modulo value is used to obtain the bit information of the corresponding bit in the BF memory. If the corresponding bit is "0", the element is not in the set, and the insertion operation is performed to set the corresponding position to "1"; if one of the corresponding bits is not "0", it is considered that the element is not in the set; Otherwise, if all corresponding positions after the element is hashed are "1", it is considered that the element may already be in the set.
8. A highly efficient billing call record re-ordering method according to claim 7, characterized in that: After receiving the target identification information, the target server performs HashMap calculation based on the Bloom filter to generate k binary numbers, and at the same time takes out k positions in the BitSet to make a judgment: If there is a scenario where one bit is 0, it is judged as a non-duplicate order. Before returning to the client, all non-0 elements in the k positions need to be updated to 1. Then the target identification information requested by the client is synchronously written to the corresponding TeleDB cluster in an asynchronous manner; If there is a scenario where the bit values of all k bits are 1, it is considered that there may be a duplicate order. After returning to the client, the client uses the keyword to search the corresponding business table in the TeleDB cluster. If the search is successful, it is considered not a duplicate order. If the search fails, it is considered a duplicate order and the corresponding record is written to the TeleDB cluster.