A method of deduplication when data without business primary key is stored in database

Abstract

The invention discloses a method for deduplicating data without a business primary key when storing a database, which is characterized in that it comprises the following steps: a data conversion module splices external business data imported into the database into character strings, and a data hash operation module splices and combines The resulting string is calculated using the sha256 algorithm to obtain a byte array. The message digest conversion module converts the message digest in byte array format into a string H1. The message digest hash module performs hash operations on the converted string H1 again to obtain a For the integer value H2, the deduplication processing module mainly uses the values ​​H1 and H2 obtained by the two hash operations in the above steps as conditions to query the database. In the present invention, the method for deduplicating the non-business primary key data when storing the database adopts the characteristic of extremely low collision rate according to the result of the message digest algorithm, and can judge whether the data are equal by only comparing two fields, and effectively utilizes the database index to Improve efficiency.

Description

technical field [0001] The invention relates to the technical field of database query deduplication, in particular to a method for deduplication of data without a business primary key when storing databases. Background technique [0002] Usually, when designing the database table structure, the business primary key field is designed, and the uniqueness of the data is judged through the business primary key field, but sometimes some externally input data does not have a business primary key, and we need to judge before storing these data Whether the same data already exists to determine the subsequent processing method. When the business primary key does not exist, it is necessary to query whether there is the same data by using each field of the data as a query condition. In this way, the amount of data in the table is very large The efficiency will be very low, especially when the stored fields are not suitable for adding database indexes. Contents of the invention [00...

AI Technical Summary

Problems solved by technology

[0002] Usually, when designing the database table structure, the business primary key field is designed, and the uniqueness of the data is judged through the business primary key field, but sometimes some externally input data does not have a business primary key, and we need to judge before storing these data Whether the same data already exists to determine the subsequent processing method. When the business primary key does not exist, it is necessary to query whether there is the same data by using each field of the data as a query condition. In this way, the amount of data in the table is very large The efficiency will be very low, especially when the stored fields are not suitable for adding database indexes

Images