Smart city data management system and method

Abstract

The invention relates to a smart city data management system and method, and the system comprises: a data collection unit which is used for integrating collection and processing technologies of DataX,flume, MaxWell, distributed crawlers and kafka, obtaining data, and carrying out the extraction, cleaning, filtering and conversion of the data, so as to obtain a processing result; a data storage unit which is used for carrying out integration and storage management on the multi-source heterogeneous data in the processing result to obtain a storage result; and a data processing unit which is used for carrying out task division on the storage result to form sub-tasks, calling different calculation rules to process the sub-tasks according to different big data processing scenes to obtain sub-processing results, and synthesizing the sub-processing results to obtain a total processing result. The smart city data management system and method have the capabilities of collecting, storing, processing and managing mass data, support linear expansion of data storage, processing and computing capabilities, and can support the whole smart city application system.

Description

Technical field [0001] The invention relates to the technical field of big data resource management, and more specifically to a smart city data management system and method. Background technique [0002] Smart cities connect real cities with digital cities through the ubiquitous Internet of Things. With the rapid development of new generation network technology, information technology and Internet of Things technology, big data supporting platform technology has become one of the important supporting technologies in the field of modern smart cities. One. At present, various government departments have a large amount of urban operation data resources. However, due to scattered management and unsound system standards, repeated collections have been caused. The caliber is too chaotic. The data has quality problems such as incomplete, incomplete, incorrect, and inconsistent data. The cost is high and the degree of sharing is poor. With the government’s transition to "economic regul...

AI Technical Summary

Problems solved by technology

For structured data, the query, statistics, and update efficiency of massive data is low. For unstructured data, such as pictures, videos, etc., it is difficult to store and retrieve files. For semi-structured data, it needs to be converted into structured data for storage and analysis. , or store according to unstructured data, which is more difficult; big data management issues, how to effectively manage distributed, multi-source, heterogeneous big data, there is currently no effective means; Combined use of multiple business data can be analyzed to obtain effective result information

Images