Cross-platform data integration method and device based on data-in-platform

Abstract

The invention provides a cross-platform data integration method and device based on a data intermediate station, and the method comprises the steps: collecting to-be-integrated target data from a specified data source to the data intermediate station according to a data extraction rule, converting the target data in various formats into a preset standard format, and guaranteeing the data consistency. According to the method, the target data converted into the preset standard format is stored in the data table in a layered manner, and the data blood relationship graph is constructed based on the target data converted into the preset standard format, so that the data integration process is simplified, and the possibility of data integration errors is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-platform data integration method and device based on a data middle platform. Background Art

[0002] Different data sources use different database types, data formats, and interface standards, which leads to difficulties in data exchange and sharing, and the inability to guarantee data consistency. The data integration process is more complicated and prone to errors. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a cross-platform data integration method and device based on a data middle platform to ensure data consistency, streamline the data integration process and reduce the possibility of data integration errors.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] A first aspect of an embodiment of the present invention discloses a cross-platform data integration method based on a data middle platform, the method comprising:

[0006] According to the pre-configured data extraction rules, the target data to be integrated is collected from the specified data source to the data center;

[0007] Converting the target data into a preset standard format;

[0008] The target data converted into the preset standard format is stored in layers in the data center;

[0009] A data lineage relationship diagram is constructed based on the target data converted into the preset standard format.

[0010] Preferably, the target data converted into the preset standard format is stored in layers in the data center, including:

[0011] Identify the storage layer of the target data converted into the preset standard format in the data middle platform, the storage layer is a business system layer, a data integration layer, a data application layer, a data source layer or a data aggregation layer;

[0012] The target data converted into the preset standard format is stored in the data center according to the corresponding storage hierarchy.

[0013] Preferably, constructing a data blood relationship diagram based on the target data converted into the preset standard format includes:

[0014] identifying a data hierarchy structure of the target data converted into the preset standard format;

[0015] constructing a data link according to the data hierarchy;

[0016] Based on the data link, a data lineage relationship diagram is constructed.

[0017] Preferably, according to the pre-configured data extraction rules, the target data to be integrated is collected from the specified data source to the data center, including:

[0018] Get the interface standard of the specified data source;

[0019] According to the interface standards and pre-configured data extraction rules, the target data to be integrated is collected from the specified data source to the data middle platform.

[0020] Preferably, before the target data converted into the preset standard format is hierarchically stored in the data center, the method further includes:

[0021] According to a pre-configured quality assessment standard, assessing the data quality of the target data converted into the preset standard format;

[0022] If the data quality is unqualified, the target data converted into the preset standard format is adjusted.

[0023] A second aspect of an embodiment of the present invention discloses a cross-platform data integration device based on a data middle platform, the device comprising:

[0024] The collection unit is used to collect the target data to be integrated from the specified data source to the data center according to the pre-configured data extraction rules;

[0025] A conversion unit, used for converting the target data into a preset standard format;

[0026] A storage unit, used for storing the target data converted into the preset standard format in a hierarchical manner in the data center;

[0027] A construction unit is used to construct a data lineage relationship diagram based on the target data converted into the preset standard format.

[0028] Preferably, the storage unit comprises:

[0029] An identification module, used to identify the storage level of the target data converted into the preset standard format in the data middle platform, wherein the storage level is a business system layer, a data integration layer, a data application layer, a data source layer or a data aggregation layer;

[0030] A storage module is used to store the target data converted into the preset standard format in the data center according to the corresponding storage level.

[0031] Preferably, the building block comprises:

[0032] An identification module, used for identifying a data hierarchy structure of the target data converted into the preset standard format;

[0033] A first building module, configured to build a data link according to the data hierarchy;

[0034] The second construction module is used to construct a data lineage relationship diagram based on the data link.

[0035] Preferably, the collection unit comprises:

[0036] The acquisition module is used to obtain the interface standard of the specified data source;

[0037] The collection module is used to collect the target data to be integrated from the specified data source to the data center according to the interface standard and pre-configured data extraction rules.

[0038] Preferably, it also includes:

[0039] An evaluation unit, configured to evaluate the data quality of the target data converted into the preset standard format according to a pre-configured quality evaluation standard;

[0040] The adjustment unit is used to adjust the target data converted into the preset standard format if the data quality is unqualified.

[0041] Based on the above-mentioned embodiment of the present invention, a cross-platform data integration method and device based on a data middle platform is provided. The method is: according to the pre-configured data extraction rules, the target data to be integrated is collected from the designated data source to the data middle platform; the target data is converted into a preset standard format; the target data converted into the preset standard format is stored in layers in the data middle platform; and a data lineage relationship diagram is constructed based on the target data converted into the preset standard format. In this solution, the target data to be integrated is collected from the designated data source to the data middle platform according to the data extraction rules, and the target data in various formats are converted into the preset standard format to ensure data consistency. The target data converted into the preset standard format is stored in layers in the data middle platform, and a data lineage relationship diagram is constructed based on the target data converted into the preset standard format, so as to streamline the data integration process and reduce the possibility of data integration errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flowchart of a cross-platform data integration method based on a data middle platform provided in an embodiment of the present invention;

[0044] Figure 2 A structural block diagram of a cross-platform data integration device based on a data middle platform provided in an embodiment of the present invention;

[0045] Figure 3 Another structural block diagram of a cross-platform data integration device based on a data middle platform provided by an embodiment of the present invention;

[0046] Figure 4 Another structural block diagram of a cross-platform data integration device based on a data middle platform provided by an embodiment of the present invention;

[0047] Figure 5 Another structural block diagram of a cross-platform data integration device based on a data middle platform provided in an embodiment of the present invention;

[0048] Figure 6 Another structural block diagram of a cross-platform data integration device based on a data middle platform provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0051] Research has found that the data center is usually used to achieve automatic collection, processing and integration management of multi-source heterogeneous data. However, different data sources use different database types, data formats and interface standards, which makes data exchange and data sharing difficult, and data consistency cannot be guaranteed. The data integration process is more complicated and prone to errors.

[0052] In order to solve the above problems, this solution proposes a cross-platform data integration method and device based on a data middle platform, which collects the target data to be integrated from the specified data source to the data middle platform according to the data extraction rules, and converts the target data in various formats into a preset standard format to ensure data consistency. The target data converted into the preset standard format is stored in layers in the data middle platform, and a data lineage relationship diagram is constructed based on the target data converted into the preset standard format, which simplifies the data integration process and reduces the possibility of data integration errors.

[0053] See also Figure 1 , shows a flow chart of a cross-platform data integration method based on a data middle station provided by an embodiment of the present invention. The cross-platform data integration method is applicable to the data middle station, and the cross-platform data integration method includes:

[0054] Step S101: According to the pre-configured data extraction rules, the target data to be integrated is collected from the specified data source to the data center.

[0055] It should be noted that the ETL (Extract-Transform-Load) tool is deployed in advance on the data middle platform. ETL is used to represent the process of extracting (Extract), transforming (Transform), and loading (Load) data from the data source to the destination.

[0056] Complete the following specified configurations in advance in the data center: configure the specified data source to be connected, configure data extraction rules, configure data conversion logic, and configure the target data to be loaded.

[0057] Specifically, the configuration interface of the data center is displayed, which includes multiple configuration items; in response to operations on multiple configuration items triggered by the user in the configuration interface, the specified data source and the target data to be loaded are determined, and the data extraction rules and data conversion logic are obtained.

[0058] That is to say, the user can operate in the configuration interface to complete the above-mentioned "specified configuration".

[0059] It should be further explained that each designated data source is a business system or a business platform.

[0060] In the specific implementation of step S101, target data to be integrated are collected from at least one designated data source according to data extraction rules, and the target data to be integrated are collected into the data center. The target data are multi-source heterogeneous data.

[0061] It is understandable that different data sources may use different interface standards, and when collecting data from a data source, the interface standard of the data source needs to be utilized.

[0062] In some embodiments, the specific implementation method of collecting target data from a specified data source to a data middle platform is: obtaining the interface standard of the specified data source; and collecting the target data to be integrated from the specified data source to the data middle platform according to the interface standard of the specified data source and pre-configured data extraction rules.

[0063] Specifically, the ETL tool deployed by the data center is called to collect target data from the specified data source to the data center according to the interface standards and data extraction rules of the specified data source.

[0064] Step S102: Convert the target data into a preset standard format.

[0065] It is understandable that the formats of target data collected from a designated data source are usually not uniform. To better integrate the target data, the formats of the collected target data need to be converted into a uniform preset standard format.

[0066] In the process of implementing step S102, the pre-configured data conversion logic is used to convert the format of the target data into a unified preset standard format. The preset standard format can be JSON format or XML format, or other formats. The specific type of the preset standard format can be determined according to actual needs.

[0067] Specifically, the data mapping and conversion engine of the data center is used to convert the format of the target data collected from the specified data source into a preset standard format.

[0068] It can be seen from the contents of the above steps S101 and S102 that the collection and processing of target data involves data transmission and communication between different ends (between the data center and the designated data source). In order to better realize data transmission and communication between different ends, this solution adopts middleware technology to realize the transmission and processing of target data.

[0069] Specifically, the message queue and data bus are used to realize the transmission and processing of target data. That is, after the target data is collected from the specified data source, the collected target data is put into the message queue, and then the format of the target data put into the message queue is converted into a preset standard format.

[0070] Through the above steps S101 and S102, automatic collection and processing of target data can be achieved, thereby achieving integration of target data.

[0071] Step S103: The target data converted into a preset standard format is stored in a hierarchical manner in the data center.

[0072] In the specific implementation of step S103, in order to meet different business needs, the target data converted into a preset standard format is hierarchically stored and managed in the data center, where the "target data converted into a preset standard format" is the "integrated target data".

[0073] Specifically, identify the storage layer of the target data converted into a preset standard format in the data middle platform, which storage layer is the business system layer, data integration layer, data application layer, data source layer or data aggregation layer.

[0074] The target data converted into a preset standard format is stored in the data center according to the corresponding storage hierarchy.

[0075] That is to say, in the data center, all "target data converted into the preset standard format" are stored and managed in layers according to the business system layer, data integration layer, data application layer, data source layer and data aggregation layer.

[0076] In actual applications, data warehouses and data marts are built on the data middle platform. Based on the built data warehouses and data marts, the target data converted into a preset standard format is stored and managed in a hierarchical manner in the data middle platform.

[0077] Step S104: constructing a data lineage relationship diagram based on the target data converted into a preset standard format.

[0078] In the process of implementing step S104, based on the data governance framework of the data center, a data lineage relationship diagram is constructed based on the target data converted into a preset standard format.

[0079] In some embodiments, the data lineage relationship diagram is constructed by: identifying the data hierarchy structure of the target data converted into a preset standard format, constructing a data link according to the data hierarchy structure; and constructing the data lineage relationship diagram based on the data link.

[0080] Specifically, according to the categories of the “target data converted into the preset standard format”, a data hierarchy structure among the “target data converted into the preset standard format” is formed.

[0081] According to the data hierarchy between the "target data converted into the preset standard format", the connection between the "target data converted into the preset standard format" is established, thereby forming a data link. According to the data link, a data lineage relationship diagram is constructed.

[0082] The data lineage relationship diagram is displayed in the specified interface of the data center, and data traceability is achieved through the data lineage relationship diagram.

[0083] In some embodiments, the data quality of the target data converted into a preset standard format is evaluated according to a preconfigured quality assessment standard; if the data quality of the "target data converted into the preset standard format" is unqualified, the target data converted into the preset standard format is adjusted (modified and reintegrated) to adjust the data quality of the "target data converted into the preset standard format" to be qualified.

[0084] On the contrary, if the data quality of "target data converted into the preset standard format" is qualified, no adjustment will be made.

[0085] Through the application of the above-mentioned "data lineage diagram" and "quality assessment standards", the accuracy, completeness and consistency of the target data can be ensured.

[0086] In the embodiment of the present invention, the target data to be integrated is collected from the designated data source to the data center according to the data extraction rules, and the target data in various formats are converted into a preset standard format to ensure data consistency. The target data converted into the preset standard format is stored in layers in the data center, and a data lineage relationship diagram is constructed based on the target data converted into the preset standard format, which simplifies the data integration process and reduces the possibility of data integration errors.

[0087] After applying this solution, on the one hand, it can achieve seamless integration and sharing of data across platforms and systems, break the "data island", and promote business collaboration and innovation; on the other hand, it improves the efficiency and accuracy of data integration, reduces manual intervention and error rates; on the third hand, it can flexibly adjust the data integration process and rules to quickly respond to business changes.

[0088] Corresponding to the cross-platform data integration method based on the data middle platform provided in the above embodiment of the present invention, see Figure 2 , the embodiment of the present invention also provides a structural block diagram of a cross-platform data integration device based on a data middle station, the cross-platform data integration device is applicable to the data middle station, the cross-platform data integration device includes: a collection unit 100, a conversion unit 200, a storage unit 300 and a construction unit 400;

[0089] The collection unit 100 is used to collect the target data to be integrated from the specified data source to the data center according to the pre-configured data extraction rules.

[0090] The conversion unit 200 is used to convert the target data into a preset standard format.

[0091] The storage unit 300 is used to store the target data converted into a preset standard format in a hierarchical manner in a data center.

[0092] The construction unit 400 is used to construct a data lineage relationship diagram based on the target data converted into a preset standard format.

[0093] In the embodiment of the present invention, the target data to be integrated is collected from the designated data source to the data center according to the data extraction rules, and the target data in various formats are converted into a preset standard format to ensure data consistency. The target data converted into the preset standard format is stored in layers in the data center, and a data lineage relationship diagram is constructed based on the target data converted into the preset standard format, which simplifies the data integration process and reduces the possibility of data integration errors.

[0094] Preferably, combined Figure 2 , see Figure 3 , shows another structural block diagram of a cross-platform data integration device based on a data middle platform provided by an embodiment of the present invention, wherein the storage unit 300 includes:

[0095] The identification module 301 is used to identify the storage level of the target data converted into a preset standard format in the data middle platform, and the storage level is the business system layer, data integration layer, data application layer, data source layer or data aggregation layer.

[0096] The storage module 302 is used to store the target data converted into a preset standard format in the data center according to the corresponding storage level.

[0097] Preferably, combined Figure 2 , see Figure 4 , shows another structural block diagram of a cross-platform data integration device based on a data middle platform provided by an embodiment of the present invention, wherein the construction unit 400 includes:

[0098] The identification module 401 is used to identify the data hierarchy structure of the target data converted into a preset standard format.

[0099] The first building module 402 is used to build a data link according to the data hierarchy structure.

[0100] The second construction module 403 is used to construct a data lineage relationship diagram according to the data link.

[0101] Preferably, combined Figure 2 , see Figure 5, shows another structural block diagram of a cross-platform data integration device based on a data middle platform provided by an embodiment of the present invention, wherein the collection unit 100 includes:

[0102] The acquisition module 101 is used to acquire the interface standard of the specified data source.

[0103] The collection module 102 is used to collect the target data to be integrated from the specified data source to the data center according to the interface standard and pre-configured data extraction rules.

[0104] Preferably, combined Figure 2 , see Figure 6 , shows another structural block diagram of a cross-platform data integration device based on a data middle platform provided by an embodiment of the present invention, the cross-platform data integration device also includes:

[0105] The evaluation unit 500 is used to evaluate the data quality of the target data converted into a preset standard format according to a pre-configured quality evaluation standard.

[0106] The adjustment unit 600 is used to adjust the target data converted into a preset standard format if the data quality is unqualified.

[0107] In summary, the embodiment of the present invention provides a cross-platform data integration method and device based on a data middle platform, which collects target data to be integrated from a specified data source to the data middle platform according to data extraction rules, and converts target data in various formats into a preset standard format to ensure data consistency. The target data converted into the preset standard format is stored in layers in the data middle platform, and a data lineage relationship diagram is constructed based on the target data converted into the preset standard format, which simplifies the data integration process and reduces the possibility of data integration errors.

[0108] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0109] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0110] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-platform data integration method based on a data middle platform, characterized in that: The method comprises: According to the pre-configured data extraction rules, the target data to be integrated is collected from the specified data source to the data center; Converting the target data into a preset standard format; The target data converted into the preset standard format is stored in layers in the data center; A data lineage relationship diagram is constructed based on the target data converted into the preset standard format.

2. The method according to claim 1, characterized in that The target data converted into the preset standard format is stored in layers in the data center, including: Identify the storage layer of the target data converted into the preset standard format in the data middle platform, the storage layer is a business system layer, a data integration layer, a data application layer, a data source layer or a data aggregation layer; The target data converted into the preset standard format is stored in the data center according to the corresponding storage hierarchy.

3. The method according to claim 1, characterized in that Constructing a data lineage relationship diagram based on the target data converted into the preset standard format includes: identifying a data hierarchy structure of the target data converted into the preset standard format; constructing a data link according to the data hierarchy; Based on the data link, a data lineage relationship diagram is constructed.

4. The method according to claim 1, characterized in that: According to the pre-configured data extraction rules, the target data to be integrated is collected from the specified data source to the data center, including: Get the interface standard of the specified data source; According to the interface standards and pre-configured data extraction rules, the target data to be integrated is collected from the specified data source to the data middle platform.

5. The method according to any one of claims 1 to 4, characterized in that: Before storing the target data converted into the preset standard format in layers in the data center, the method further includes: According to a pre-configured quality assessment standard, assessing the data quality of the target data converted into the preset standard format; If the data quality is unqualified, the target data converted into the preset standard format is adjusted.

6. A cross-platform data integration device based on a data middle platform, characterized in that: The device comprises: The collection unit is used to collect the target data to be integrated from the specified data source to the data center according to the pre-configured data extraction rules; A conversion unit, used for converting the target data into a preset standard format; A storage unit, used for storing the target data converted into the preset standard format in a hierarchical manner in the data center; A construction unit is used to construct a data lineage relationship diagram based on the target data converted into the preset standard format.

7. The device according to claim 6, characterized in that The storage unit comprises: An identification module, used to identify the storage level of the target data converted into the preset standard format in the data middle platform, wherein the storage level is a business system layer, a data integration layer, a data application layer, a data source layer or a data aggregation layer; A storage module is used to store the target data converted into the preset standard format in the data center according to the corresponding storage level.

8. The device according to claim 6, characterized in that The building block comprises: An identification module, used for identifying a data hierarchy structure of the target data converted into the preset standard format; A first building module, configured to build a data link according to the data hierarchy; The second construction module is used to construct a data lineage relationship diagram based on the data link.

9. The device according to claim 6, characterized in that The acquisition unit comprises: The acquisition module is used to obtain the interface standard of the specified data source; The acquisition module is used to collect the target data to be integrated from the specified data source to the data center according to the interface standard and pre-configured data extraction rules.

10. The device according to any one of claims 6 to 9, characterized in that: Also includes: An evaluation unit, configured to evaluate the data quality of the target data converted into the preset standard format according to a pre-configured quality evaluation standard; The adjustment unit is used to adjust the target data converted into the preset standard format if the data quality is unqualified.

Images