Method for eliminating repetition of memory dialog list

Abstract

The invention relates to a method for eliminating repetition of memory dialog list, comprising the following steps of: step 1, reading a dialog list document into a memory; step 2, reading a dialog list record from the dialog list document; step 3, finding the index list which is corresponding to the dialog list record from the memory according to the key information of the dialog list record; step 4, piecing the field content of the dialog list record to form a character string, working out the MD5 value as the index of the dialog list record; step 5, interpolating the MD5 value in the index list; if the interpolating operation is successful, writing the dialog list record into a normal dialog list document; otherwise, writing the dialog list record into a repeated list document; and step 6, repeating the second step to the fifth step until traversing all dialog list records of the dialog list document. The method provided by the invention eliminates the repetition by using the index list and the MD5 value, realizes the operation for implementing relatively quick repetition eliminating treatment in a relatively small memory space, and the repetition eliminating processing performance of the dialog list is not relative to the data volume of the dialog list.

Description

technical field [0001] The invention relates to a deduplication mode of call list records, in particular to a method for solving the memory level deduplication of massive files in a telecommunication system. Background technique [0002] With the continuous development of telecom operators, the business expansion is wider and wider, and various new services emerge in endlessly; as the tariff becomes cheaper and cheaper, the number of users continues to grow, and the number of calls of users is also increasing year by year. getting bigger. The original de-duplication needle is getting more and more massive data bills, the pressure is getting bigger and bigger, and the system upgrade is imperative. At present, most telecom operators adopt the following methods of database, memory and file system. The deduplication products implemented by these methods have more or less defects in performance and IO in massive data processing. [0003] Database method: The general practice of...

AI Technical Summary

Problems solved by technology

This method needs to be searched in the whole file, which is related to the performance and file size. The performance is still possible if the number of bills is small. If the number of bills is large, the IO operation will increase and affect the performance.

[0005] Pure-memory method: Run the duplicate check files in the memory. This method does not rely on the functions provided by the database, and the processing speed is fast, but the number of duplicate check files is large. In addition to occupying part of the hard disk storage resources, it also requires system There are relatively large memory resources, and there are relatively strict requirements for the host memory when processing a large number of bills

Images