Data integration method based on data center

Abstract

The invention discloses a data integration method based on a data center. The method is based on data integration construction requirements of a multi-source heterogeneous data system, and comprises the following steps: firstly, establishing a data standard, constructing a data model, and establishing a unified specification for multi-source heterogeneous data in the system; building a data warehouse based on the data model, and storing data in the integration process; meanwhile, a data center is constructed, and data services such as data extraction, conversion, loading and display are provided by relying on an ETL technology and a data object encapsulation access method; and finally, distributing and synchronizing the integrated data by using a message middleware technology. Data integration of the multi-source heterogeneous data system can be achieved, and the method has the advantages of being high in stability, low in coupling, high in availability and the like.

Description

Technical Field

[0001] The present invention relates to the field of data integration, and in particular to a data integration method based on a data center. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] Since large-scale integrated business information platforms usually contain many subsystems, each subsystem has different business divisions and is developed independently. Due to factors such as development time and team, there is usually a lack of unified standards and planning, resulting in large differences in application development technology and data structure. As the pace of system informatization construction accelerates, the problem of "information islands" where information exchange between business systems is not smooth will arise. Although each business system has independent functions, there are business process connections between systems. With the development of informatization construction, more attention is paid to information exchange between business systems, and the demand for building a multi-source heterogeneous data integration system to achieve information integration between business systems is increasing day by day.

[0004] The implementation method of data integration is closely related to the characteristics of the business itself. Each system needs to design the integration method according to its own situation. There are mainly the following methods for data integration at home and abroad: The first is to realize information integration based on XML data exchange technology (Liu Yang. Development of information integration platform based on data exchange technology [J]. Information Technology and Informatization, 2021, 12(038)); The second is to use SOAP message mechanism and Web Service technology (Wei Jie. Research on e-government data exchange technology based on Web Service [D]. Guangxi University, 2018); The third is to use the exchange front-end method to exchange in the form of library tables and files through the front-end node. The main technology uses ETL tools to transpose and load the data records in the database into the target server (Wang Xiaobo. Application of data exchange platform based on ETL technology in the field of e-government [J]. Computer Knowledge and Technology, 2016, 12(21): 278-280). The first method requires the use of XML format or JSON to encapsulate data when transmitting data. For structured data originally stored in the database, this operation increases the data processing process and will affect data integration performance. The second method requires coding in the original system, which makes significant changes to the original system. The third method is the ETL technology in data center services, but traditional data center-based data integration methods also have problems with weak data maintainability and consistency. Summary of the invention

[0005] The purpose of the present invention is to provide a data center-based data integration method to address the above technical problems, add data object encapsulation access technology for other businesses to obtain data, and use middleware technology to distribute the integrated data in real time to achieve data integration between multi-source heterogeneous systems.

[0006] The technical solution to achieve the purpose of the present invention is: a data center-based data integration method, the steps are as follows:

[0007] Step 1: Establish data standards and build data models. Since data between business systems is multi-source and heterogeneous, a unified data structure and standard must be established to achieve data integration, so that the data in the integration process is based on a unified description specification.

[0008] Step 2: Build a data warehouse to store data during the multi-source heterogeneous system integration process.

[0009] Step 3: Build a data center to provide a variety of data services, including using ETL technology and data object encapsulation access methods to implement data extraction, conversion, loading and display services.

[0010] Step 4: Distribute the data in the data center based on the message middleware technology to synchronize the data among multi-source heterogeneous systems.

[0011] Furthermore, in step 1, the data standard is established and the steps for building the data model are as follows:

[0012] Step 1-1, establish data standards. First, classify the business system data that needs to be integrated. This method only supports the integration of structured data in the business system. The structured data in the system has the characteristics of multi-source heterogeneity. When building data standards, it is necessary to keep the structure of similar data as consistent as possible, including table composition, table name, field type, etc., to facilitate the establishment of data mapping relationships during later data extraction.

[0013] Step 1-2: Create data models for corresponding data tables based on established data standards, including data logical models and data physical models, to prepare for building a data warehouse.

[0014] Furthermore, the steps for building a data warehouse in step 2 are as follows:

[0015] Step 2-1: The data warehouse is used to store the data processed in the integration method and consists of a business entity library and a metadata library.

[0016] Step 2-2: The business-related data physical model constructed in step 1 will be built in the business entity library, and the metadata physical model constructed will be built in the metadata library.

[0017] Furthermore, the steps for building a data center in step 3 are as follows:

[0018] Step 3-1: Use JAVA language, based on Vue+SpringBoot+Mybatis framework, to build a data center according to the data warehouse obtained in step 2.

[0019] Step 3-2: The data center implements data services based on ETL technology, and the data services include data extraction, data conversion, data loading and data display.

[0020] Furthermore, the data service step in step 3-2 includes:

[0021] The data extraction service realizes its function by extracting the multi-source data of the system into a temporary database;

[0022] The data conversion service realizes its function by mapping the extracted data according to the established data conversion rules;

[0023] The data loading service realizes its function by storing the transformed data into the data warehouse;

[0024] The data display service implements its functions by encapsulating access methods through data objects.

[0025] Data extraction, conversion, and loading services are all based on the Kettle tool in ETL technology. The Kettle tool is integrated into the data center service framework to implement the data ETL process.

[0026] Furthermore, the data object encapsulation access method includes:

[0027] Data model encapsulation, defines the data object model in the data warehouse, and establishes the mapping relationship between the object model and the database table object.

[0028] Data access service publishing: on the server side, the data access service is developed and configured through the application framework, and then packaged and deployed to the server to publish the data access service.

[0029] Process data access requests. Other business systems in the system obtain data through data object encapsulation access interfaces. After receiving the request, the server performs corresponding database operations and returns the result data packet. The requester deserializes the data packet after receiving it.

[0030] Furthermore, the steps for distributing data based on the message middleware technology in step 4 are as follows:

[0031] Step 4-1: Use message middleware to implement data distribution. Create a message producer in the data center service built in step 3. Each business system creates a message consumer. The producer distributes data to consumers.

[0032] Step 4-2: Create a message topic based on the business so that consumers can obtain integrated business data and achieve business data synchronization.

[0033] Furthermore, the message middleware adopts Kafka message middleware.

[0034] Furthermore, the data center is developed based on the Vue+SpringBoot+Mybatis framework.

[0035] Compared with the prior art, the present invention has the following significant advantages: the present method adopts SOA architecture, provides data integration methods by building data center services, and is decoupled from the integrated business systems. The present method realizes efficient data extraction, conversion and loading by integrating mature Kettle tools, thereby ensuring the integration of multi-source heterogeneous data in the system. The present method adopts data object encapsulation access methods for each business system to obtain data, thereby ensuring the security of the data warehouse. The present method uses middleware technology to realize the distribution of integrated data. Compared with traditional technologies, the present method has higher integration performance than using XML data exchange technology to realize data integration; it makes less changes to the original system than using SOAP message mechanism and Web Service technology to realize data integration, and is more decoupled; it has more data maintenance and sharing functions than using a single ETL technology to realize data integration, and is more versatile. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0037] Figure 1 It is the overall flow chart of the present invention.

[0038] Figure 2 It is an overview diagram of the implementation process of the present invention.

[0039] Figure 3 It is the method to create a data warehouse implementation diagram.

[0040] Figure 4 It is the database data conversion relationship diagram in the method.

[0041] Figure 5 It is a diagram of the data center service composition in the method.

[0042] Figure 6 It is the implementation diagram of the data object encapsulation access method in the method.

[0043] Figure 7 This is the implementation diagram of the message distribution function in the method. DETAILED DESCRIPTION

[0044] like Figure 1As shown, the present invention provides a data integration method based on a data center, adds data object encapsulation access technology for other businesses to obtain data, and uses middleware technology to distribute the integrated data in real time to achieve data integration between multi-source heterogeneous systems.

[0045] Combination Figure 2 The present invention provides a data integration method based on a data center, and the specific implementation steps are as follows:

[0046] Step 1: Establish data standards, build data models, and generate data warehouses, such as Figure 3 The implementation process is described in detail below:

[0047] (1) Sorting data into categories: This method is only for multi-source heterogeneous data in the system. First, the business system data is sorted according to functional classification. For example, if there are functional divisions such as personnel, operation and maintenance, and core business in the system, the data will be divided into personnel data, operation and maintenance data, and core function data.

[0048] (2) Organize data structure: Establish unified standards for data, so as to organize the logical model and physical model of each business data, such as Figure 4 As shown, for example, to perform statistics on personnel data tables from multiple data sources, the integrated personnel table should contain the basic fields required by all personnel tables in the system. The field names should be as unified as possible, the fields should not be redundant, and the metadata information of the personnel table should be recorded at the same time.

[0049] (3) Build a data warehouse: According to the purpose of data, it is divided into a business entity database and a metadata database. The business entity database contains the users and tables required by each business system, and the metadata database stores information describing the data of each business system. Implement the sorted data model, that is, execute the table creation statement to generate a data warehouse.

[0050] Step 2: Build a data center, such as Figure 5 As shown, the data center provides a variety of data services, including data extraction, data conversion, data loading and data display. The specific implementation process is as follows:

[0051] (1) Building a data center service framework: The present invention adopts SOA architecture, uses JAVA language, is developed based on Vue+SpringBoot+Mybatis framework, and integrates Kettle tool.

[0052] (2) Write ETL methods to implement various data services: First, add Kettle dependencies to the pom file in the project to ensure that the project can use Kettle's ETL function library, then use the KettleEnvironment class to initialize the Kettle environment to load the necessary components and configurations, then use the TransMeta class to read the data source configuration, configure data mapping, and finally use the Trans class to perform data conversion, ultimately achieving data extraction, data conversion, and loading.

[0053] (3) Use data objects to encapsulate access methods: Figure 6 As shown, the data object encapsulation access method in the data center service is mainly used for data display. The implementation steps are data model encapsulation, data access service publishing, and data access request processing. The specific implementation method is as follows:

[0054] Data model encapsulation: define the data object model in the data warehouse, map the object model with the database table object, define database operation statements based on the table, such as add, delete, modify, and query operations, and use the Protobuf serialization framework to carry out data transmission between users and services.

[0055] Data access service release: The data center service is based on the SpringBoot framework and is packaged using Maven. The jar package is started directly on the server to release the data access service.

[0056] Processing data access requests: If other business systems in the system need to access the data in the data warehouse, they can obtain the data through the interface encapsulated by the data object. The interface is based on the Http protocol. After obtaining the request, the service uses the Mybatis framework to perform database operations and uses SpringMVC to return the access request. After receiving the request, the requester needs to use the proto file to deserialize the data packet.

[0057] Step 3: Implement data distribution in data centers based on the message middleware Kafka, such as Figure 7 The specific implementation process is as follows:

[0058] (1) Formulate message topics: According to the data type, formulate topics for each type of data that needs to be distributed, which are used for communication between message producers and message consumers, such as personnel management topics, core business topics, system operation and maintenance topics, and metadata topics.

[0059] (2) Create a message producer: The message producer belongs to the data center service and is responsible for distributing the data integrated in the data warehouse to various business systems. The data center adds Kafka dependencies to the service's pom file and uses the send method of the KafkaTemplate class to send messages containing data.

[0060] Create message consumers: Message consumers belong to various business systems and are responsible for receiving data distributed by the data center to achieve data integration of multi-source heterogeneous systems. By adding Kafka dependencies to the service, you can use the KafkaListener class to receive messages from the corresponding topics. For example, create a personnel management consumer in the personnel management business system to receive personnel management Topic messages, and create a core business consumer in the core business business system to receive core business Topic messages.

[0061] The present invention provides a method and idea for a data integration method based on a data center. There are many methods and approaches to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A data integration method based on a data center, characterized in that: The following steps are involved: Step 1: Establish data standards and build data models according to requirements; Step 2: Build a data warehouse and generate database tables based on the data model obtained in step 1; Step 3: Build a data center and implement data services based on ETL technology and data object encapsulation access methods; Step 4: Distribute the data in the data warehouse based on message middleware technology to achieve data synchronization between multi-source heterogeneous systems.

2. The data integration method based on data center according to claim 1, characterized in that: In step 1, data standards are established and the steps for building a data model are as follows: Step 1-1, establish data standards, classify the system data of the target business, and build data standards for the classified system data. The data standards include table composition, table name and field type; Step 1-2: Create a data model corresponding to the data table according to the data standard established in step 1-1, wherein the data model includes a data logical model and a data physical model.

3. The data integration method based on data center according to claim 2, characterized in that: The steps to build a data warehouse in step 2 include: Step 2-1: Define the business entity library and metadata library according to business requirements; Step 2-2: construct the data physical model obtained in step 1 in the business entity library, and construct the metadata physical model in the metadata library.

4. The data integration method based on data center according to claim 3 is characterized in that: The steps to build a data center in step 3 include: Step 3-1, build a data center based on the data warehouse obtained in step 2; Step 3-2: The data center implements data services based on ETL technology, and the data services include data extraction, data conversion, data loading and data display.

5. The data integration method based on data center according to claim 4, characterized in that: The steps to implement data services in step 3-2 include: The data extraction service realizes its function by extracting the multi-source data of the system into a temporary database; The data conversion service realizes its function by mapping the extracted data according to the established data conversion rules; The data loading service realizes its function by storing the transformed data into the data warehouse; The data display service implements its functions by encapsulating access methods through data objects.

6. The data integration method based on data center according to claim 5, characterized in that: The data extraction, data conversion and data loading services are all implemented based on the Kettle tool in ETL technology.

7. The data center-based data integration method according to claim 6, characterized in that: The data object encapsulation access method includes: Data model encapsulation, defining the data object model in the data warehouse and establishing the mapping relationship between the object model and the database table object; Data access service publishing: on the server side, the data access service is developed and configured through the application framework, and then packaged and deployed to the server to publish the data access service externally; Process data access requests. Other business systems in the system obtain data through data object encapsulation access interfaces. After receiving the request, the server performs corresponding database operations and returns the result data packet. The requester deserializes the data packet after receiving it.

8. The data integration method based on data center according to claim 7, characterized in that: In step 4, the steps for distributing data based on message middleware technology are as follows: Step 4-1: Data distribution is implemented based on the message middleware. A message producer is created in the data center service constructed in step 3. Each business system creates a message consumer. The message producer distributes data to the message consumer. Step 4-2: Create a message topic based on the data type so that consumers can obtain integrated business data and achieve business data synchronization.

9. The data center-based data integration method according to claim 8, characterized in that: The message middleware adopts Kafka message middleware.

10. The data integration method based on data center according to claim 1, characterized in that: The data center is developed based on the Vue+SpringBoot+Mybatis framework.

Images