Data conversion method of heterogeneous data source and middleware

A technology of heterogeneous data sources and conversion methods, applied in structured data retrieval, electronic digital data processing, database management systems, etc., can solve the problems of insufficient types of heterogeneous data sources, difficulty in automatic conversion, and failure to support domestic databases, etc.

Pending Publication Date: 2020-07-28
CHINA POWER IND INTERNET CO LTD
View PDF2 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Aiming at the problems that the existing data conversion methods support scenarios, the types of heterogeneous data sources are insufficient, domestic databases are not supported, and it is difficult to realize automatic conversion, etc., the present invention provides a data conversion method and middleware for heterogeneous data sources, which support multiple structures Structured, semi-structured and unstructured data sources, supporting four domestic databases of Wuhan Dameng, Renda Jincang, Shenzhou GM, and NTU GM; using process engine technology to realize personalized control of the data conversion process; through task scheduling, to achieve Automated Data Transformation

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data conversion method of heterogeneous data source and middleware
  • Data conversion method of heterogeneous data source and middleware
  • Data conversion method of heterogeneous data source and middleware

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0084] image 3 Embodiment 1 of the present invention provides a flow chart of a data conversion method for heterogeneous data sources. This example takes the core data conversion function as an example. The method specifically includes the following:

[0085] Step 101, configure the access data source, and obtain the technical metadata of the access data source.

[0086] According to the different types of different access data sources, the configuration information is different, as follows:

[0087] Database-type data sources need to configure the specific type of database, database address, user name, password, database instance name (can be empty), and driver package. The specific type of database such as Dameng, Oracle, MySQL, MongoDB, etc.; the database address such as the IP address or domain name address of the database server; the user name and password are the authorization to access the database; if there are multiple instances of the database, one needs to be spe...

Embodiment 2

[0115] In this embodiment, on the basis of the first embodiment above, a process engine is introduced to streamline the data conversion process.

[0116] The data conversion process includes six core nodes: access-exit source configuration, data access adaptation, data acquisition, processing conversion, data access adaptation, and data loading, such as Figure 4 shown.

[0117] Node 201, access-access source configuration.

[0118] The main task of this node is to select the access data source and the access data source, and configure the access-exit mapping relationship. The content of the configuration is the same as step 103 in the first embodiment. If the access data source and the output data source have not been registered as data sources, they need to be registered in advance. The content of the registration information is the same as that in steps 101-102 of the first embodiment.

[0119] Node 202, data access adaptation.

[0120] The main task of this node is to ...

Embodiment 3

[0130] On the basis of the second embodiment above, this embodiment introduces task scheduling to automate the data conversion process.

[0131] Configurable scheduling strategy, supports real-time task scheduling, scheduled task scheduling, and supports setting priorities. Scheduling tasks can be associated with one or more data conversion processes, and support parallel and sequential execution of multiple data conversion processes, such as Figure 5 a, 5b, 5c.

[0132] Figure 5 a indicates the sequential scheduling of multiple data conversion processes, that is, the combined scheduling of some data conversion processes has a sequence, and the input of the latter conversion process depends on the output of the previous conversion process, and must be executed in a certain order.

[0133] Figure 5 b indicates parallel scheduling of multiple data conversion processes, that is, the input of a certain conversion process depends on the output of multiple conversion processes...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data conversion method for a heterogeneous data source and middleware. The method comprises the following steps: 1, adding source data and target data, and configuring a datasource type and connection information for the source data and the target data; step 2, establishing a data conversion process, wherein the data conversion process comprises data access adaptation, data collection, processing conversion and data output adaptation which are in functional series connection; 3, configuring a data conversion scheduling strategy of the heterogeneous data source basedon a data conversion process in a task scheduling form; and step 4, executing data conversion of the heterogeneous data source number according to the data conversion scheduling strategy. The method supports various structured, semi-structured and non-structured data sources, and supports four domestic databases of Wuhan Dream, Huajinbao, Shenzhou universality and South Day universality; a processengine technology is adopted to realize personalized control of a data conversion process; and through task scheduling, automatic data conversion is realized.

Description

technical field [0001] The invention relates to the technical field of data integration and governance, in particular to a data conversion method and middleware for heterogeneous data sources. Background technique [0002] With the development of enterprise informatization, in response to the needs of business departments, the information center department assisted in the construction of business systems such as HR, finance, production, and sales. However, on the one hand, due to the lack of information planning and top-level design, no requirements were put forward for the database when constructing / purchasing business systems. Different business systems adopted heterogeneous or different versions of database data sources, and some business systems developed Http interface data sources; On the one hand, there are some core businesses that are not supported by business systems, and the data is transferred in offline forms such as spreadsheets; at the same time, there are mul...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25G06F9/54
CPCG06F16/258G06F9/546G06F2209/547
Inventor 杨瑛余艺韩李琦
Owner CHINA POWER IND INTERNET CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap