Data management method and device

Abstract

The embodiments of the present application provide a data management method and device. The method first receives a data access request from a data center station for raw data access, and determines multiple candidates corresponding to the raw data according to a data source identifier carried in the data access request. client, and create multiple target sub-threads and multiple target sub-loaders corresponding to the multiple candidate clients, and then call the corresponding target sub-loaders through the multiple target sub-threads, respectively, to load the multiple candidate clients. The candidate driver file is used to start multiple candidate clients, and then read the original data through the multiple candidate clients respectively to obtain multiple candidate data, determine the target data according to the multiple candidate data, and finally connect the target data to the data center. In this application, multiple sub-threads and sub-loaders are created to respectively access data from different sources to the data center, thereby improving the data management efficiency.

Description

technical field [0001] The present application relates to the technical field of data management, and in particular, to a data management method and device. Background technique [0002] In the era of big data, more and more data information is flooding people's work and life. In order to facilitate the management of data, researchers have introduced the platform of data center, where users can unify data from different sources in the data center. manage. [0003] However, in the process of connecting data to the data center, users find that data from the same source and different versions often conflict during the process of accessing the data center. For example, the open source kafka component and the kafka component of Huawei's big data platform, If you want to access the data center at the same time, there is a code package conflict between them, that is to say, the open source kafka data and Huawei's kafka data cannot be connected to the data center at the same time. ...

AI Technical Summary

Problems solved by technology

[0003] However, in the process of connecting data to the data center, users find that different versions of data from the same source often conflict in the process of accessing the data center. For example, the open source Kafka component and the Kafka component of Huawei's big data platform, If you want to access the data center at the same time, they have code package conflicts, that is to say, open source kafka data and Huawei's kafka data cannot be connected to the data center at the same time

The traditional solution is to develop corresponding small tools to convert different versions of data from the same source into data of the same version through the small tools, and then access the data center. This traditional data access method requires first Develop small tools, so the data management efficiency is too low, and the system development cost is high

Images