Data cleaning and indexing method

Abstract

The invention discloses a data cleaning and indexing method, belonging to the field of data retrieval. The data cleaning and indexing method comprises the following specific processes of: (1), cleaning and importing data: cleaning structured data, and cleaning unstructured data; (2), establishing metadata: 1, obtaining the metadata, integrating different metadata in multiple sources of a data centre through a metadata obtaining process, and performing unified storage management by using a database as a metadata knowledge base; 2, publishing the metadata, and managing publishing of the metadata by establishing a set of metadata publishing process; and 3, accessing the metadata, creating a set of metadata access permission conferring and managing process mechanism, and controlling effective access of a valid user to metadata data; and (3), constructing an index: obtaining data attributes on a distributed data cluster of the data centre through metadata access. The overall data attributes are very clear; the structured and unstructured data are distinguished definitely; for different data structural types, a cleaning tool can be selected in a targeted manner; therefore, the best cleaning conversion effect can be realized; and the data integration conversion efficiency can be greatly increased.

Description

technical field [0001] The invention belongs to the field of data retrieval, and in particular relates to a data cleaning and indexing method based on a cloud platform multi-dimensional data fusion analysis system. Background technique [0002] At present, in terms of enterprise platform construction, it is mainly based on multi-party data, such as manually imported data, including txt format, excel format, csv format, etc., synchronized data, data obtained from other business systems in real time, etc., through The analysis of the data, the directional summary of the data, for later retrieval, but with the continuous expansion of the scale of the enterprise, the continuous growth of the business volume, the multi-party data generated by itself is also increasing sharply, and the traditional data is used for the general data. method or purchase a ready-made retrieval method to perform data cleaning and retrieval. However, most of the data comes from various platforms or the...

AI Technical Summary

Problems solved by technology

However, most of the data comes from various platforms or the summary of data from all parties. Since these data are not organized and organized before, all the data is in a fragmented state. If you still use ordinary data collection, data cleaning , the way of data retrieval, since these retrieval tools are ready-made tools, it is impossible to further improve and modify the tools, so basically all these data cannot be summarized on the basis of existing retrieval tools. , which has caused a lot of trouble and a waste of resources for the enterprise, because only by trying on the basis of the existing retrieval methods can it be found that the content cannot be used or the retrieval effect is not good. This process itself is a waste of resources. process, so continue to make targeted improvements