Method for duplicate removal of service-free primary key data during database storage

Abstract

The invention discloses a method for duplicate removal of service-free primary key data during database storage. The method comprises the following steps: the data conversion module splices the external service data imported into the database into a character string; the data hash operation module calculates the spliced and combined character strings by using a sha256 algorithm to obtain a byte array; the message digest conversion module converts the message digest in the byte array format into a character string H1; the message digest hash module performs hash operation on the character string H1 converted from the message digest again to obtain an integer value H2, and the duplicate removal processing module queries a database mainly by taking the values H1 and H2 obtained by two times of hash operation in the above steps as conditions. According to the method for duplicate removal of service-free primary key data during database storage, whether the data are equal or not can be judged by comparing only two fields according to the characteristic that the collision rate of a message digest algorithm result is extremely low, and the database index is effectively utilized to improvethe efficiency.

Description

technical field [0001] The invention relates to the technical field of database query deduplication, in particular to a method for deduplication of data without a business primary key when storing databases. Background technique [0002] Usually, when designing the database table structure, the business primary key field is designed, and the uniqueness of the data is judged through the business primary key field, but sometimes some externally input data does not have a business primary key, and we need to judge before storing these data Whether the same data already exists to determine the subsequent processing method. When the business primary key does not exist, it is necessary to query whether there is the same data by using each field of the data as a query condition. In this way, the amount of data in the table is very large The efficiency will be very low, especially when the stored fields are not suitable for adding database indexes. Contents of the invention [00...

AI Technical Summary

Problems solved by technology

[0002] Usually, when designing the database table structure, the business primary key field is designed, and the uniqueness of the data is judged through the business primary key field, but sometimes some externally input data does not have a business primary key, and we need to judge before storing these data Whether the same data already exists to determine the subsequent processing method. When the business primary key does not exist, it is necessary to query whether there is the same data by using each field of the data as a query condition. In this way, the amount of data in the table is very large The efficiency will be very low, especially when the stored fields are not suitable for adding database indexes

Images