Heterogeneous data source standardized processing method and device and server

Abstract

The embodiment of the invention provides a heterogeneous data source standardized processing method and device and a server. An information index of a universal structure can be generated according tosummary information of a to-be-processed data source; then the to-be-processed heterogeneous data source is divided into a plurality of data source fragments, and then the data source fragments are converted into data set blocks under a preset operation framework according to area information indexes of the data source fragments; finally, the data set blocks are integrated to obtain a standardized data set. Compared with the prior art, the data source fragments are directly obtained from the to-be-processed data source, and therefore it is guaranteed that the acquired data source is timely and accurate; the data source fragments are converted into the data set blocks through cluster nodes of the server in a multi-thread mode, and the data set blocks are integrated into the standardized data set, so that data processing time is greatly shortened, and the real-time performance of data is guaranteed; besides, when the content in the data source changes, only the changed data source fragments need to be re-read and converted, so that timely updating of the data set is achieved.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a heterogeneous data normalization processing method, device and server. Background technique [0002] With the advent of the information age, the generation and replacement of information data is accelerating, and the amount of information data is also increasing rapidly. In the large amount of information data, due to the different sources of data sources, the data types and data structures are also various. Since data sources with different data structures use different analysis logics, when performing data analysis on multiple data sources with different data structures, it is not possible to directly use general analysis logics for unified processing of these data sources. [0003] In the prior art, in order to achieve unified processing of heterogeneous data, a preprocessing method for heterogeneous data sources is used, such as figure 1 As shown, th...

AI Technical Summary

Problems solved by technology

However, due to the huge amount of data in the data source, it takes a lot of time to preprocess each time, and the timeliness of the data in the database cannot be guaranteed.

Moreover, for data analysis, valuable data only accounts for a part of the data source, therefore, the methods in the prior art will process and store a large amount of worthless data, resulting in a waste of hardware storage resources

[0005] Moreover, since the data content, configuration information, and data structure of the data source will change at any time, in the process of data analysis, in order to ensure the timely and accurate data analysis, it is necessary to load the latest data source in real time. Preprocessing takes a lot of time, and the data source in the database can only be updated once at a certain interval. When performing data analysis, the data source stored in the database may have expired. The latest data source is obtained, so there is no way to guarantee the real-time and accuracy of the data

Images