Data storage method, electronic device and machine readable storage medium

Abstract

The embodiment of the invention provides a data storage method, an electronic device and a machine readable storage medium. The method comprises: the electronic device acquiring first data needing tobe written into the data table, obtaining a fingerprint calculation template corresponding to a data source of the first data, and calculating a first fingerprint of the first data according to the fingerprint calculation template and the first data, searching a second fingerprint of the second data from the data source from the duplicate checking table according to the data source, judging whether the searched second fingerprint is the same as the first fingerprint or not, and deleting the first data if the searched second fingerprint is the same as the first fingerprint. Thus, rapid duplicate checking, duplicate removal and storage of the data can be realized based on the duplicate checking table. In the duplicate checking process, the data table does not need to be queried. Indexes do not need to be established for different data sources on the data table to increase the speed of querying the data table. The problem that the writing performance of the data table is poor due to the fact that multiple indexes on the data table are dynamically maintained is solved, and the physical space occupied by the indexes on the data table and the maintenance expenditure are saved.

Description

technical field [0001] The present application relates to the technical field of data storage, and in particular, to a data storage method, electronic equipment, and a machine-readable storage medium. Background technique [0002] In the current data analysis system, due to basic software or data acquisition and other reasons, there are often duplicate data among different batches of data, which will have a negative impact on the authenticity of the data analysis results and the system visualization effect. [0003] Therefore, it is necessary to check the obtained data and the data written in the data table first. In order to improve the efficiency of finding data from the data table, the data that needs to be read can be quickly found from the data table by building an index on the data table for comparison with the data to be checked. However, when the data in the data table changes, the index established on the data table needs to be dynamically updated, and subsequent w...

AI Technical Summary

Problems solved by technology

[0004] If the data in the data table is obtained from multiple data sources, the fields for judging whether the data of different data sources are repeated are different, and it is necessary to index one or more fields for each data source to judge whether the data is repeated, resulting in the data table When writing data, it is necessary to dynamically update multiple indexes, which takes a long time, resulting in very slow data writing speed in the data table

[0005] In addition, in addition to the data table, the index needs to occupy a certain amount of physical space, and it takes time to maintain, which increases the storage burden and maintenance overhead of the database.

Images