Message deduplication method and device based on hash algorithm, equipment and storage medium

Abstract

The application discloses a message duplicate elimination method and device based on a hash algorithm, a computing device and a storage medium. According to the technical scheme provided by the application, a hash value and a message serial number extracted from a charging message are received by a receiving client; the hash value is calculated according to a session ID and a user number extracted from the charging message by the client; a corresponding memory database record is found from a memory database according to the message serial number and the hash value, and the charging message is eliminated according to the finding result. Through the application, the message serial number and the hash value obtained from the charging message are used to find the record in the memory database, the elimination is completed, the high performance of the elimination, the small storage space occupation, the dynamic expansion and the elimination accuracy are ensured, and the efficiency of the message elimination is greatly improved.

Claims

1. A message deduplication method based on a hash algorithm, comprising: Receive the hash value sent by the client and the message sequence number extracted from the billing message; The hash value is calculated based on the session ID and user number extracted by the client from the billing message; The corresponding memory database record is searched from the memory database according to the message sequence number and hash value, and the billing message is deduplicated based on the search results; The step of searching for the corresponding memory database record in the memory database based on the message sequence number and hash value, and then deduplicating the billing message based on the search result, further includes: Based on the message sequence number, calculate the sub-record number corresponding to the memory database record and the bit information sequence number in the memory database record; wherein, the sub-record number is the quotient of the message sequence number divided by the number of messages that can be stored in one memory database record plus 1; the bit information sequence number is the remainder of the message sequence number divided by the number of messages that can be stored in one memory database record. Based on the hash value and the record number, search the memory database to see if a corresponding memory database record exists. If no corresponding memory database record is found, the billing message is determined to be a new message, and a new memory database record is added to the memory database according to the hash value, the sub-record number, and the bit information sequence number.

2. The method according to claim 1, wherein, The in-memory database record includes: a hash value part, a message storage part, and a record number part; The step of adding a new memory database record to the memory database based on the hash value, the sub-record number, and the bit information sequence number further includes: Set the hash value of the new memory database record to the hash value, set the sub-record number of the new memory database record to the sub-record number, and set the data bit corresponding to the bit information sequence number in the message storage part of the new memory database record to the first value.

3. The method according to claim 1, wherein, The step of searching for the corresponding memory database record in the memory database based on the message sequence number and hash value, and then deduplicating the billing message based on the search result, further includes: If a corresponding memory database record is found, it is determined whether the value of the data bit corresponding to the bit information sequence number in the message storage part of the memory database record is the first value; If the value of the data bit corresponding to the bit information sequence number is a first value, the billing message is determined to be a duplicate message; if the value of the data bit corresponding to the bit information sequence number is a second value, the billing message is determined to be a new message, and the value of the data bit corresponding to the bit information sequence number is set to the first value.

4. The method according to claim 1, wherein, After searching for the corresponding memory database record in the memory database based on the message sequence number and hash value, and deduplicating the billing message according to the search result, the method further includes: The deduplication results are returned to the client.

5. A message deduplication device based on a hash algorithm, comprising: The module consists of a receiving module, a deduplication detection module, and a storage module; among which, The receiving module is used to receive a hash value sent by the client and a message sequence number extracted from the billing message; the hash value is calculated based on the session ID and user number extracted by the client from the billing message; The deduplication judgment module is used to search for the corresponding memory database record in the memory database according to the message sequence number and hash value, and to deduplicat the billing message according to the search result. The deduplication determination module is further configured to: calculate the sub-record number corresponding to the memory database record and the bit information sequence number in the memory database record based on the message sequence number; wherein, the sub-record number is the quotient of the message sequence number divided by the number of messages that can be stored in one memory database record plus 1; the bit information sequence number is the remainder of the message sequence number divided by the number of messages that can be stored in one memory database record; search for the existence of a corresponding memory database record in the memory database based on the hash value and the sub-record number; if no corresponding memory database record is found, determine that the billing message is a new message; The storage module is used to: if no corresponding memory database record is found, add a new memory database record to the memory database according to the hash value, the sub-record number, and the bit information sequence number.

6. The apparatus according to claim 5, wherein, The de-weighting determination module is further used for: If a corresponding memory database record is found, it is determined whether the value of the data bit corresponding to the bit information sequence number in the message storage part of the memory database record is the first value; If the value of the data bit corresponding to the bit information sequence number is a first value, the billing message is determined to be a duplicate message; if the value of the data bit corresponding to the bit information sequence number is a second value, the billing message is determined to be a new message, and the value of the data bit corresponding to the bit information sequence number is set to the first value.

7. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the message deduplication method based on the hash algorithm as described in any one of claims 1-4.

8. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the message deduplication method based on a hash algorithm as described in any one of claims 1-4.

Images