Safe and reliable multi-party data deduplication system, method and device

Abstract

The embodiment of the invention provides a safe and reliable multi-party data deduplication system, method and device, which are applied to the field of data processing, and the system comprises a data processing end and at least two data providing ends. And the data processing end is used for sending a data request for requesting the object data of the target object to each data providing end. And the data providing end is used for receiving the data request sent by the data processing end. And the object data of the target object is obtained in the locally stored object data. And conversionprocessing is performed on the object data by adopting a preset data conversion mode to obtain converted object data. And the converted object data is sent to a data processing end. The data processing end is used for receiving the converted object data sent by each data providing end; and repeated data in the received converted object data is determined to obtain a repeated data set. And other data in each repeated data group is removed. By applying the scheme provided by the embodiment of the invention, repeated data in the object data can be removed.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a safe and reliable multi-party data deduplication system, method and device. Background technique [0002] With the development of network technology, the network can provide more and more object data of objects such as users, vehicles, and videos. The data processing end can process the object data of each object to obtain data processing results, thereby realizing object classification, object functions such as data forecasting. However, different object data of the same object may be stored in different servers in different scenarios, and each server serves as a data provider. For example, in the case where the above-mentioned object is a user, the user’s object data such as credit card usage information, consumer shopping information, and mobile phone call information related to the user’s credit are stored in the bank’s bank server, the e-commerce server of the e-c...

AI Technical Summary

Problems solved by technology

If the data processing end directly processes the acquired object data, repeated data may affect the accuracy of data processing results

For example, if the data processing end needs to count the object data of each data provider, in the case of duplicate data, the same object data may be counted multiple times, resulting in inaccurate statistical results

Images