Multivariate heterogeneous data integration method about smart city

Abstract

The multivariate heterogeneous data integration method about the smart city comprises the following steps that S1, data sources are configured and connected, and a user carries out database configuration according to different data sources; s2, accessing a data acquisition standard module, and performing automatic data processing and verification; and S3, writing the data into a data warehouse of the big data platform through the platform integration task. The invention provides a multivariate heterogeneous data integration method for a smart city. The method can perform semi-automatic data cleaning and perfect data standards according to pre-collected data. For a smart city, the smart city relates to many industries, and the storage of the same type of data of different data sources can be merged, classified and unified through the method. Clean and normative data are provided for subsequent big data processing analysis, artificial intelligence training data and the like, and analysis accuracy and algorithm fitting are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart city, and particularly relates to a multi-element heterogeneous data integration method for smart city. BACKGROUND

[0002] The smart city is a key infrastructure component and service of city management, education, medical treatment, real estate, transportation, public utilities and public safety, etc. by the application of intelligent computing technologies such as Internet of Things, cloud computing, big data, spatial geographic information integration, etc. to make the city more interconnected, efficient and intelligent. The most bottom and primary realization of the smart city lies in the data collection of various facility components. The traditional way collects data according to the collection tools supported by different facilities, and imports data into the big data platform. This contains structured data and unstructured data, two-dimensional data and three-dimensional data, etc. The standards and storage methods of data are different.

[0003] In order to realize the data collection of the smart city, due to the huge difference between different applications and devices, the current data collection process is only a simple and rough import and storage through the matching collection tool, and when there are many data sources, a lot of time will be spent on secondary data processing. The same data may be different in format, remarks, data length, naming, data type, etc. due to different sources, although many big data platforms have special data standard functions to solve these problems, but the processing flow is too much, causing a waste of many resources and time. SUMMARY

[0004] To solve the above problems, the technical scheme provides a multi-element heterogeneous data integration method for smart city.

[0005] To achieve the above purpose, the technical scheme is as follows: A multi-element heterogeneous data integration method for smart city, comprising the following steps S1: configure data source and connect, the user configures the database according to different data sources; S2: access data collection standard module, automatic data processing and verification; wherein, comprising the following steps: S201: establish data standard library, formulate standard mapping relationship table, and finally form standard set; Among them, for structured data, the standard mapping relationship table is used to output the data collected from different data sources but belonging to the same nature to a unified expression method; for unstructured data, the file name is standardized and modified; S202: after the generation of multiple standard sets, the same items are merged, and the standard set iteration is carried out based on this; S3: write data into the data warehouse of the big data platform through platform integration task.

[0006] The method for integrating multi-element heterogeneous data of a smart city as described above, in step S1, the configuration method for structured data includes the following steps S101: log in to the big data platform through the software user login system; S102: create a new data source after entering the data source configuration page, wherein the database configuration mode includes at least one of Mysql, Oracle, Sql Server and SAP Hana.

[0007] The method for integrating multi-element heterogeneous data of a smart city as described above, in step S1, the configuration method for unstructured data includes the following steps S103: log in to the big data platform through the software user login system; S104: upload files to the big data platform for data storage and recording, and generate a file path to store data.

[0008] The method for integrating multi-element heterogeneous data of a smart city as described above, the standard mapping relationship table at least contains standard column, mapping column, date.

[0009] The method for integrating multi-element heterogeneous data of a smart city as described above, after step 2, a data quality report is generated, which details the results of data quality monitoring and detection, as well as the process of data access and standardization, and is displayed in the integrated sub-page in the form of a blood relationship map.

[0010] The method for integrating multi-element heterogeneous data of a smart city as described above, the following steps are for structured data: Step 1: use the administrator user to create a new user, authorize to create project permissions, new task permissions, new data source and target source database permissions, Hadoop cluster HDFS file system access permissions, Hive library table creation permissions, and access permissions for Web page components Hue interacting with the cluster; Step 2: the user creates a new project, creates a connection method for the database source library and the target library; Step 3: perform data exploration on the data source to understand the data type, table fields, field type, field meaning, industry, and product category of the source data; Step 4: determine whether the field is of a similar type on the platform, if so, perform hierarchical classification according to the existing data specification on the platform, if not, design a new field type and create new standards and specifications; Step 5: use the platform's automated audit module to perform preliminary automated audit on the new standards and specifications, including punctuation, characters, and length errors, and then perform manual audit through the page, correct problems, or form new field standards and specifications. Step 6: data integration task creation and configuration is performed; the specific steps are as follows: Enter the task configuration page, select the configured database connection, and select the method of using the integration standardization module for data integration; Step 7: The created task is scheduled in the form of a workflow according to actual conditions, which is divided into real-time and offline. Offline sets the daily task scheduling time, and real-time only needs to start the task.

[0011] The above-mentioned multi-element heterogeneous data integration method of a smart city, the following steps are for unstructured data: Step 1: Use the administrator user to create a new user, authorize to create project permissions, new task permissions, new data source and target source database permissions, Hadoop cluster HDFS file system access permissions, Hive library table creation permissions, and access permissions of the Web page component Hue for interacting with the cluster; Step 2: The user creates a new project, creates a database source library and a target library connection method; Step 3: Data exploration of the source file is performed to understand the data content of the source file; Step 4: Determine whether the file name is similar to the platform, if so, classify and mark according to the existing file name specification of the platform, if not, design a new file name, and create a new file name standard and specification; Step 5: Use the platform automatic audit module to perform preliminary automatic audit on the new standard and specification, the audit content is punctuation, character obvious error finding, and then perform manual audit through the page, if there is a problem, correct it, if there is no problem, form a new file name standard and specification; Step 6: file import task creation and configuration is performed, and the specific steps are as follows: Determine whether the file name standard specification operation is needed, if not, add a time marker to the file, if so, fill in the new file name in the new file name field and add a time marker at the same time; Step 7: Import the file, the file import is a one-time task, and there is no need to set a task scheduling period.

[0012] The above-mentioned multi-element heterogeneous data integration method of a smart city deploys an open source Hadoop cluster to support the functions of entire data storage, task running, and basic permission control.

[0013] The above-mentioned multi-element heterogeneous data integration method of a smart city integrates a data collection standard specification module function, performs automatic and manual audit data standard functions, and contains a visual page.

[0014] The present application has the following beneficial effects: The application provides a multi-element heterogeneous data integration method for a smart city, which can perform semi-automatic data cleaning and perfect data standard functions according to pre-acquired data. Although the initial work is more and human intervention is more, subsequent data cleaning and standard specification will gradually reduce human factors with time accumulation. For the smart city, the smart city involves many industries, and the storage of the same type of data from different data sources will be combined, classified and unified through this method. For subsequent data processing and analysis of big data, artificial intelligence training data, etc., provide relatively clean and standardized data, increase the analysis accuracy and algorithm fitting. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced as follows.

[0016] Figure 1 is a flow chart of the data integration method in the embodiments of the application; Figure 2 is a data acquisition architecture diagram in the embodiments of the application. DETAILED DESCRIPTION

[0017] In order to make the technical problems, technical solutions and beneficial effects solved by the application more clearly understood, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0018] A smart city multi-element heterogeneous data integration method, comprising the following steps S1: configure and connect the data source, and the user configures the database according to different data sources; the user configures the database according to different data sources, different database configuration methods are different, but the same as the traditional connection method. Containing the ip, port, access account and password of the data source and other information; the platform contains login permission control, different users can access and view different databases after login, and the administrator has the permission of all library tables and data acquisition tasks.

[0019] S2: access the data acquisition standard module and perform automatic data processing and verification; wherein, comprising the following steps: S201: establish a data standard library, develop a standard mapping relationship table, and finally form a standard set; Among them, for structured data, the standard mapping relationship table is used to output the data collected from different data sources but belonging to the same nature to a unified expression method; for unstructured data, the file name is standardized and modified; S202: after the generation of a plurality of standard sets, the same items are combined, and the standard set iteration is performed based on this; The working principle of step S2 is that a data collection standard module needs to be accessed before data collection, which is available to all users, and the most important point of the technical solution is that the data is first processed and verified by the perfect collection standard before being imported into the big data platform, so that data of different data sources but the same nature can be classified and unified. During the process, data classification, unified identification and other data processing methods are used.

[0020] For example, the table imported from A data source contains information of person B, including age, ID card and other data. The table imported from B data source also contains the age and ID card of person B, but A data source stores the age by number, such as 30 years old, while B data source stores the age by text, such as thirty. The age information of the two data sources represents the same data, but due to different storage methods and business sources, the same data is stored in completely different formats. The collection standard can reduce or completely avoid such situations. The specific method is as follows: First, a standard data standard library is designed. The design source mainly refers to national standards and government or company regulations, such as international standards or head company standard documents if there is no reference, or self-developed standards if there is no standard reference. According to the actual business situation, habits and other methods, a standard library is established, and a mapping relationship of different texts is established, such as using numerical values to express age. If there are texts or other expressions, they are converted through the standard mapping relationship table. The standard mapping relationship table includes standard columns, mapping columns, dates, etc. The format is generally as follows: Standard column Mapping column Date 30 Thirty 2025.5.1 That is, any text used to store age will be automatically converted to a numerical value. This method is suitable for a combination of automatic and manual methods, and requires manual maintenance of standard modules and corresponding relationships. The standards established at the beginning will not be complete, but through manual review and gradual improvement of the standard module, the manpower required for subsequent data unification will gradually decrease. This method is suitable for industries with strong specificity, such as hospitals and education, and a single industry can be further subdivided. A single industry or industry subdivision will eventually form a standard set.

[0021] After multiple standard sets are generated, the same items are merged, and the standard set is iterated based on this. When the standard set is gradually improved, most of the collected data can be standardized during the collection process, greatly reducing the workload of subsequent data processing.

[0022] For unstructured data such as audio, video, documents, etc. since the content cannot be directly modified, only the file name needs to be standardized, such as non-standard file naming, file name containing spaces, non-standard date, characters, symbols, etc. Before collection, first understand the file content and meaning, and then rename, classify according to the industry, so that the corresponding file can be quickly retrieved during use.

[0023] S3: Write data into the data warehouse of the big data platform through platform integration tasks. The tasks are based on projects, and different roles can enter the projects according to role permissions for allocation and use. The integration tasks include various data integration methods: wizard mode, script mode, API mode, file transfer, file parsing, and whole library migration.

[0024] Further, as a preferred embodiment of the present scheme but not limited, in step S1, the configuration method of structured data includes the following steps S101: Log in to the big data platform through the software user login system; S102: After entering the data source configuration page, create a new data source, wherein the database configuration mode includes at least one of Mysql, Oracle, Sql Server and SAP Hana. Most of them are structured data, including two-dimensional and three-dimensional data formats.

[0025] Further, as a preferred embodiment of the present scheme but not limited, in step S1, the configuration method of unstructured data includes the following steps S103: Log in to the big data platform through the software user login system; S104: Upload files to the big data platform for data storage and recording, and generate file paths to store data.

[0026] Further, as a preferred embodiment of the present scheme but not limited, the standard mapping relationship table at least includes standard columns, mapping columns, and dates.

[0027] Further, as a preferred embodiment of the present scheme but not limited, a data quality report is generated after step 2, which details the results of data quality monitoring and detection, as well as the process of data access and standardization, and displays the blood relationship graph in the integrated sub-page in a visual manner. The data provenance can be clearly understood through the visual chart.

[0028] All the above processes will save the process state through log recording and record the running situation of the data quality management module, including data quality detection, cleaning and visualization process, and visualization results. The audit function can record and track data access, modification and operation, etc. to ensure data compliance and security.

[0029] Further, as a preferred embodiment of the present scheme but not limited, the following steps are for structured data: Step 1: Create a new user using an administrator user, authorize to create project permissions, new task permissions, new data source and target source database permissions, Hadoop cluster HDFS file system access permissions, Hive library table creation permissions, access permissions of Web page components Hue interacting with the cluster; Step 2: The user creates a new project, creates a connection method of the database source library and the target library; Step 3: Data exploration of the data source is performed to understand the data types of the source data, various table fields, field types, field meanings, industries, and product categories; Step 4: It is judged whether the field is of a similar type on the platform. If so, hierarchical classification is performed according to the existing data specification on the platform. If not, a new field type is designed, and a new standard and specification are created. For example, the file name is "Patent Technology Disclosure - Software - Copy", and "- Copy" is the non-standard file name content that needs to be removed. Different files have different non-standard phenomena.

[0030] Step 5: The new standard and specification are preliminarily audited using the platform automatic auditing module, and obvious errors such as punctuation, characters, and length are found. Then, manual auditing is performed through the page. If there is a problem, it is corrected. If there is no problem, a new field standard and specification are formed; Step 6: Data integration task creation and configuration are performed. The specific steps are as follows: Enter the task configuration page, select the configured database connection, and select the data integration method using the integration standardization module; Step 7: The created task is scheduled in the form of a workflow according to the actual situation. It is divided into real-time and offline. Offline sets the daily task scheduling time, and real-time only needs to start the task.

[0031] Further, as a preferred embodiment of the present scheme but not limited, the following steps are for unstructured data: Step 1: Create a new user using an administrator user, authorize to create project permissions, new task permissions, new data source and target source database permissions, Hadoop cluster HDFS file system access permissions, Hive library table creation permissions, access permissions of Web page components Hue interacting with the cluster; Step 2: The user creates a new project, creates a connection method of the database source library and the target library; Step 3: Data exploration of the source file is performed to understand the data content of the source file; Step 4: Determine whether the file name is similar to the platform, if so, according to the platform file name specification classification mark, if not, design new file name, and new file name standard and specification; Step 5: Use the platform automation audit module to conduct preliminary automated audit on the new standard and specification, and find obvious errors such as punctuation and characters. Then, through the page, manual audit is conducted. If there is a problem, it is corrected. If there is no problem, new file name standard and specification are formed. Step 6: Create and configure file import task, and the specific steps are as follows: Determine whether the file name standard specification operation is needed. If not, add time mark to the file. If yes, fill in the new file name in the new file name field and add time mark at the same time. Step 7: Import the file. File import is a one-time task and does not need to set the task scheduling period.

[0032] Further, as a preferred embodiment of the present scheme, not limited, deploy open source Hadoop cluster to support the functions of entire data storage, task running and basic permission control.

[0033] Further, as a preferred embodiment of the present scheme, not limited, integrate data collection standard specification module function, conduct automatic and manual audit data standard function, and contain visual page. It is the most important module of the entire integrated method.

[0034] At present, data governance or standard processing is based on the work done after data import. The difference of this method is that it can be pre-cleaned in the integration process, and the subsequent data cleaning and standard work can be saved for more standardized data.

[0035] The above only describes the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Other principles and basic structures similar to the present application are within the scope of the present application.

Claims

1. A method for integrating diverse and heterogeneous data in smart cities, characterized in that: Includes the following steps S1: Configure and connect to the data source; users configure the database according to different data sources. S2: Connect to the data acquisition standard module for automated data processing and verification; this includes the following steps: S201: Establish a data standard library, formulate a standard mapping relationship table, and finally form a standard set; Specifically, for structured data, a standard mapping table is used to ensure that data collected from different data sources but belonging to the same nature are output in a unified manner; for unstructured data, file names are standardized and modified. S202: After multiple standard sets are generated, similar items are merged, and standard set iteration is performed based on this. S3: Write data into the data warehouse of the big data platform through platform integration tasks.

2. The method for integrating diverse and heterogeneous data in a smart city according to claim 1, characterized in that: In step S1, the method for configuring structured data includes the following steps: S101: Log in to the big data platform through the software user login system; S102: After entering the data source configuration page, create a new data source. The database configuration method includes at least one of MySQL, Oracle, SQL Server, and SAP Hana.

3. The method for integrating diverse and heterogeneous data in a smart city according to claim 1, characterized in that: In step S1, the method for configuring unstructured data includes the following steps: S103: Log in to the big data platform through the software user login system; S104: Upload files to the big data platform for data storage and recording, and generate file path storage data.

4. The method for integrating diverse and heterogeneous data in a smart city according to claim 1, characterized in that: A standard mapping table must contain at least a standard column, a mapping column, and a date.

5. The method for integrating diverse and heterogeneous data in a smart city according to claim 1, characterized in that: After step 2, a data quality report is generated, which records in detail the results of data quality monitoring and testing, as well as the data access and standardization process. The report is displayed in an integrated subpage as a kinship map, presented in a visual manner.

6. The method for integrating diverse and heterogeneous data in a smart city according to claim 1, characterized in that: The following steps are for structured data: Step 1: Create a new user using the administrator user and grant permissions for creating projects, creating new tasks, creating new data sources and target source databases, access to the Hadoop cluster HDFS file system, creating databases and tables in Hive, and access to the Hue web page component for interacting with the cluster. Step 2: The user creates a new project and sets the connection method between the source and target databases; Step 3: Conduct data exploration of the data source to understand the data type of the source data, the fields of each table, the field type, the meaning of the field, the industry to which it belongs, and the product category to which it belongs; Step 4: Determine if there are similar categories of fields on the platform. If so, classify them according to the existing data specifications of the platform. If not, design new field types and create new standards and specifications. Step 5: Use the platform's automated review module to conduct an initial automated review of the new standards and specifications. Review the content such as punctuation, characters, and length errors to find obvious errors. Then, conduct a manual review through the page. If there are any problems, correct them. If there are no problems, form new field standards and specifications. Step 6: Create and configure the data integration task; The specific steps are as follows: Enter the task configuration page, select the configured database connection, and choose to integrate the data using the integrated standardized module method; Step 7: Schedule the created tasks in the form of a workflow according to the actual situation. This is divided into real-time and offline. For offline tasks, set the daily task scheduling time. For real-time tasks, simply start the task.

7. The method for integrating diverse heterogeneous data in a smart city according to claim 1, characterized in that: The following steps are for unstructured data: Step 1: Create a new user using the administrator user and grant permissions for creating projects, creating new tasks, creating new data sources and target source databases, access to the Hadoop cluster HDFS file system, creating databases and tables in Hive, and access to the Hue web page component for interacting with the cluster. Step 2: The user creates a new project and sets the connection method between the source and target databases; Step 3: Probe the source file to understand its data content; Step 4: Determine if there are similar file names on the platform. If so, classify and label them according to the platform's existing file name specifications. If not, design new file names and create new file name standards and specifications. Step 5: Use the platform's automated review module to conduct an initial automated review of the new standards and specifications, such as checking for obvious errors in punctuation and characters. Then, conduct a manual review through the page. If there are any problems, make corrections; if there are no problems, form new file name standards and specifications. Step 6: Create and configure the file import task. The specific steps are as follows: Determine whether standard file name formatting is required. If not, add a timestamp to the file. If so, enter the new file name in the new file name field and add the timestamp at the same time. Step 7: Simply import the file. File import is a one-time task and there is no need to set a task scheduling cycle.

8. The method for integrating diverse and heterogeneous data in a smart city according to claim 1, characterized in that: Deploy an open-source Hadoop cluster to support the entire data storage, task execution, and basic access control functions.

9. The method for integrating diverse and heterogeneous data in a smart city according to claim 1, characterized in that: It integrates data collection standards and specifications modules, enabling automated and manual review of data standards, and includes a visual interface.

Images