Blood relationship data management method and device based on data warehouse link and computer equipment

By establishing synchronous tasks and collecting metadata on the digital warehouse link, establishing blood relationship links, the problem of difficult to manage blood relationships between different types of digital warehouses is solved, and data traceability and governance efficiency are improved.

CN118939710APending Publication Date: 2024-11-12BEIJING QUANTUM SONG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410920336.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art cannot effectively establish blood relationships between different types of digital warehouses, and there are errors in blood collection based on passive monitoring, so blood collection of multiple digital warehouse instances cannot be achieved.

Method used

By establishing a synchronization task for business data, using synchronization tools to synchronize business data from source instances to target instances, collecting basic metadata and synchronizing configuration information, recording mapping relationships, establishing blood relationships, and realizing blood relationships between different types of data warehouses manage.

Benefits of technology

The blood relationship management between different types of data warehouses is realized, which ensures the traceability and transparency of data and improves the efficiency of data governance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939710A_ABST
    Figure CN118939710A_ABST
Patent Text Reader

Abstract

The invention relates to the field of data management, in particular to a consanguinity data management method and device based on a data warehouse link and computer equipment. The method comprises the following steps: establishing a synchronization task of business data according to business requirements, and synchronizing the business data from a source instance to a target instance by using a synchronization tool; metadata of a source instance and synchronization configuration information of the synchronization tool are collected respectively, and the synchronization configuration information records basic information of a target table configured in a target instance and a mapping relation between the source instance and the target instance in the business data synchronization process of the synchronization tool; according to the basic information, matching detailed information of a target table from basic metadata of the source instance; and establishing a blood relationship link between the service end and the target instance according to the mapping relationship and the detailed information of the target table. According to the specification, by recording the blood relationship of the data, tracing the data flow and the data conversion process, the traceability of the data is ensured, potential data quality problems are found, and the data governance efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to the field of data governance, especially to lineage data management methods, devices and computer equipment based on data warehouse links. Background Art

[0002] There are currently two main types of data lineage collection solutions in the industry: Based on the open source Apache Atlas, Atlas provides metadata management and governance services. Enterprises can build catalogs for their own Hadoop ecosystem data assets, classify and govern them. The main functions include: metadata classification, metadata retrieval, and lineage dependency (table to table, field to field).

[0003] Major Internet companies also provide many solutions and products for data governance and data lineage. However, since the basic technical architectures adopted by each company are different, they are generally developed in a targeted manner according to the technical stack of their own data middle platform, and it is difficult to form a universal data lineage system. The Apache Atlas commonly used in the industry currently uses passive monitoring technology to achieve hiveSql lineage collection. This crawling method has certain errors. For example, daily test records will also be collected. In addition, Atlas cannot achieve lineage collection of other data warehouse instances such as doris, flink, mysql, kafka, and cannot establish lineage between different types of data warehouses, such as the lineage relationship between hive tables and doris tables, BI dashboards, and API interfaces. Summary of the invention

[0004] In order to solve the problem that the above-mentioned prior art cannot establish a blood relationship between different types of data warehouses, the embodiments of this specification provide a blood relationship data management method, device and computer equipment based on a data warehouse link.

[0005] An embodiment of the present specification provides a lineage data management method based on a data warehouse link, the method comprising: establishing a business data synchronization task according to business needs, and using a synchronization tool to synchronize business data from a source instance to a target instance; respectively collecting basic metadata of the source instance and synchronization configuration information of the synchronization tool, the synchronization configuration information recording the basic information of the target table configured in the target instance by the synchronization tool during the business data synchronization process and the mapping relationship between the source instance and the target instance; matching detailed information of the target table from the basic metadata of the source instance according to the basic information; establishing a lineage link between the business end and the target instance according to the mapping relationship, the basic metadata of the source instance and the detailed information of the target table.

[0006] According to one aspect of an embodiment of the present specification, using a synchronization tool to synchronize business data from a source instance to a target instance includes: generating a target table in the target instance library based on the target instance library in the synchronization task and the data source table in the source instance, wherein the target table corresponds to the data source table; and transferring the business data in the data source table to the target table.

[0007] According to one aspect of an embodiment of the present specification, collecting basic metadata of each instance also includes: regularly collecting basic metadata in the source instance and the target instance according to a pre-period, wherein the basic metadata is used to mark the information of each storage unit in each instance; comparing the basic metadata collected for the last two times to determine whether the basic metadata of each instance has changed; if so, updating the basic metadata of each instance.

[0008] According to one aspect of an embodiment of the present specification, according to the basic information, matching the detailed information of the target table from the basic metadata of the source instance includes: determining the data source table and the target table according to the mapping relationship between the source instance and the target instance in the synchronization tool; and querying the detailed information of the target table from the basic metadata of the data source table according to the table name of the target table.

[0009] According to one aspect of an embodiment of the present specification, the synchronization configuration information further includes: basic information of the first target table configured in the first target instance, and a mapping relationship among the source instance, the target instance and the first target instance. According to the mapping relationship, the basic metadata of the source instance and the detailed information of the target table, establishing a bloodline link between the business end and the target instance further includes: matching the detailed information of the first target table in the first target instance from the basic metadata of the source instance according to the basic information of the first target instance recorded in the synchronization configuration information; establishing a bloodline link between the business end and the target instance according to the mapping relationship, the basic metadata of the source instance, the detailed information of the target table in the target instance and the detailed information of the first target table of the first target instance.

[0010] According to one aspect of an embodiment of the present specification, the source instance and the target instance are deployed in either the same business department or different business departments.

[0011] According to one aspect of an embodiment of the present specification, if an abnormality occurs in data in a bloodline link, the cause of the abnormality is determined based on basic metadata of the data.

[0012] An embodiment of the present specification provides a lineage data management device based on a data warehouse link, the device comprising: a synchronization unit, used to establish a synchronization task for business data according to business needs, and use a synchronization tool to synchronize business data from a source instance to a target instance; a collection unit, used to respectively collect basic metadata of the source instance and synchronization configuration information of the synchronization tool, the synchronization configuration information recording the basic information of the target table configured in the target instance by the synchronization tool during the business data synchronization process and the mapping relationship between the source instance and the target instance; a matching unit, used to match detailed information of the target table from the basic metadata of the source instance according to the basic information; a lineage link establishment unit, used to establish a lineage link between the business end and the target instance according to the mapping relationship, the basic metadata of the source instance and the detailed information of the target table.

[0013] An embodiment of the present specification provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the bloodline data management method based on a data warehouse link when executing the computer program.

[0014] The embodiments of this specification also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the bloodline data management method based on the data warehouse link.

[0015] This manual records the data's lineage, traces the data flow and data conversion process, ensures data traceability, discovers potential data quality issues, and improves data governance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 The figure shows a flow chart of a bloodline data management method based on a data warehouse link according to an embodiment of the present specification;

[0018] Figure 2 The figure is a flow chart of a method for synchronizing business data from a source instance to a target instance according to an embodiment of the present specification;

[0019] Figure 3 The figure is a flow chart of a method for updating basic metadata according to an embodiment of the present specification;

[0020] Figure 4The figure is a flow chart of a method for querying detailed information of a target table according to an embodiment of the present specification;

[0021] Figure 5 The figure is a flow chart of a method for establishing a blood relationship link according to an embodiment of the present specification;

[0022] Figure 6 The figure shows a schematic diagram of the structure of a bloodline data management device based on a data warehouse link according to an embodiment of the present specification;

[0023] Figure 7 The figure shows a specific structural diagram of a bloodline data management device based on a data warehouse link according to an embodiment of the present specification;

[0024] Figure 8 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification.

[0025] Description of the accompanying symbols:

[0026] 601, synchronization unit;

[0027] 602, collection unit;

[0028] 6021, comparison module;

[0029] 6022, update module;

[0030] 603, matching unit;

[0031] 604, bloodline link establishment unit;

[0032] 802. Computer equipment;

[0033] 804, processor;

[0034] 806. Memory;

[0035] 808, driving mechanism;

[0036] 810, input / output module;

[0037] 812. Input device;

[0038] 814. Output device;

[0039] 816. Presentation equipment;

[0040] 818. Graphical user interface;

[0041] 820, network interface;

[0042] 822, communication link;

[0043] 824. Communication bus. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0045] It should be noted that the terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0046] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.

[0047] It should be noted that the lineage data management method based on data warehouse links in this specification can be used in the field of data governance, as well as in the field of database or software development. This specification does not limit the application field of the lineage data management method and device based on data warehouse links.

[0048] Figure 1 The flowchart of a method for managing bloodline data based on a data warehouse link in an embodiment of this specification is shown, which specifically includes the following steps:

[0049] Step 101: Establish a business data synchronization task according to business requirements, and use a synchronization tool to synchronize business data from a source instance to a target instance.

[0050] In the embodiments of this specification, the business demand may be raised by the business department, and the business demand may be to obtain the calculation and statistical results generated by a certain type or multiple types of data in the normal operation cycle, or to analyze the causes of abnormalities in the operation of a certain type or multiple types of data. Therefore, the synchronization task of business data established according to the business demand includes: obtaining the calculation or analysis results of a specific type of business data, specifically including: what type of business data is synchronized from what instance to what instance.

[0051] However, business data is usually stored in the database of the business department, and the business department does not have the ability to make complex plans for the data. Therefore, in order to meet business needs, it is necessary to synchronize business data from one instance to another instance for specific operations and analysis. Among them, the instance storing business data can be called the source instance, and the instance opposite to the source instance can be called the target instance.

[0052] In the embodiments of this specification, the source instance may be an instance of MySQL, Kafka, etc., which is used to store business data in the upstream of the business, and the target instance may be a Hive data warehouse or a Doris data warehouse, etc., which is in the downstream big data department of the business. In some other embodiments of this specification, the source instance and the target instance may be instances belonging to the same department, for example, the source instance may be a Hive data warehouse, and the target instance may be a Doris data warehouse, etc. This specification does not limit the specific types of the source instance and the target instance.

[0053] Step 102, respectively collect the basic metadata of the source instance and the synchronization configuration information of the synchronization tool, wherein the synchronization configuration information records the basic information of the target table configured in the target instance by the synchronization tool during the business data synchronization process and the mapping relationship between the source instance and the target instance.

[0054] In the embodiments of this specification, in order to synchronize the business data in the source instance to the target instance, the DolphinScheduler task scheduling system can be used to capture the input and input table information in the synchronization tasks related to the business needs, thereby establishing the blood relationship between the tables and realizing the blood collection within each data warehouse.

[0055] Among them, the synchronization task may involve one or more tasks to synchronize business data from the source instance to the target instance. Therefore, when performing data synchronization, one or more sets of mapping relationships between source instances and target instances can be determined according to the synchronization configuration information of the synchronization tool. Each set of mapping relationships includes a source instance and a target instance.

[0056] For example, the synchronization configuration information 1 of the synchronization tool includes four groups of mapping relationships, namely: the mapping relationship between the MySQL instance and the Hive data warehouse, the mapping relationship between the Hive data warehouse and the Doris data warehouse, the mapping relationship between the Doris data warehouse and the gateway API, and the mapping relationship between the gateway API and the BI dashboard. The synchronization configuration information 1 involves four groups of mapping relationships. For another example, the synchronization configuration information 2 of the synchronization tool includes two mapping relationships, namely: the mapping relationship between the Kafka instance and the Hive data warehouse, and the mapping relationship between the Hive data warehouse and the BI dashboard.

[0057] In the embodiment of the present specification, after the data synchronization in step 101 is executed, the basic metadata of each instance involved in the data synchronization process can be collected separately. Basic metadata includes: data description information, data attribute information, data source, data update frequency, data management personnel, etc. In this step, the metadata of the source instance is collected. Specifically, the basic information of all libraries, tables, and fields of the Hive data warehouse can be obtained by calling the HiveMetaStore Thrift interface of Hive, thereby realizing the collection of basic metadata of Hive entities. The basic metadata collection of the Doris instance is obtained by accessing the tables, columns and other tables under the information_schema database that comes with Doris. The basic metadata of the Kafka topic is obtained by calling the AdminClient API interface of Kafka. The basic metadata of the BI dashboard or gateway API entity is obtained respectively through the system corresponding to the BI dashboard or gateway API.

[0058] Step 103, according to the basic information, match the detailed information of the target table from the basic metadata of the source instance. In this step, the basic metadata of the source instance includes the library table information in the source instance, including: how many libraries exist in the source instance, how many tables are included in each library, table data of each table, etc. Among them, the table data includes but is not limited to: the table name of the table in the source instance, the specific fields in the table, and other information.

[0059] According to the synchronization configuration of the synchronization tool in step 102, the target table in the target instance has a corresponding relationship with one or more tables in the source instance. The basic metadata of one or more tables in the source instance can also be used as the basic metadata of the target table in the target instance. Therefore, the basic metadata of the corresponding table can be matched from the basic metadata of multiple tables of the source instance according to the table name or other identification information of the target table. The basic metadata of the corresponding table is further used as the detailed information of the target table.

[0060] In the embodiments of the present specification, the basic metadata of each instance can be displayed in the page metadata details to improve the data visualization.

[0061] Step 104, establish a lineage link between the business end and the target instance based on the mapping relationship, the basic metadata of the source instance and the detailed information of the target table. In this step, the association relationship between different types of instances is determined based on the mapping relationship in the synchronization configuration information of the synchronization tool, and the lineage link is obtained by combining the basic metadata of the source instance and the detailed information of the target table in each set of mapping relationships.

[0062] According to the embodiment of the number of mapping relationships respectively contained in the synchronization configuration information 1 and the synchronization configuration information 2 of the synchronization tool in step 102, this specification can eventually form a data lineage link between the MySQL instance, the Hive data warehouse, the Doris data warehouse and the BI dashboard; it can also form a lineage link between the Kafka instance, the Hive data warehouse and the BI dashboard. In addition, this specification can establish a lineage relationship between the Hive data warehouse table, the Doris data warehouse table and the BI dashboard, and establish a link relationship between the lineage link and the BI dashboard, thereby realizing the function of visualizing the lineage link, displaying information such as the lineage link, metadata and metadata update data, and ensuring the traceability and transparency of data flow.

[0063] This manual can open up the entire chain of lineage, complete the comprehensive recording and tracking of data lineage, and ensure the traceability and transparency of data flow. If data in the lineage chain is abnormal, the cause of the abnormality can be determined based on the basic metadata of the abnormal data.

[0064] Figure 2 The flowchart of a method for synchronizing business data from a source instance to a target instance in an embodiment of the present specification is shown, which specifically includes the following steps:

[0065] Step 201: Generate a target table in the target instance library according to the target instance library in the synchronization task and the data source table in the source instance, wherein the target table corresponds to the data source table.

[0066] In this step, the synchronization task established according to business needs includes: what type of business data is synchronized from what instance to what instance. Among them, the target instance library is the destination instance for data synchronization, and the source instance is the instance where the business data is originally stored. Specifically, the table in the source instance where this type of business data exists can be determined from the synchronization task, and this table can be called the data source table. For example, the source instance is mysql, and the data source table in the source instance is the user table in library A in the source instance mysql. For example, the target instance library is library A in the hive instance. The synchronization task requires that the user table of library A in the source instance mysql be synchronized to library A in the hive instance.

[0067] This step is performed by the data synchronization tool. According to the target instance library in the synchronization task and the data source table in the source instance, a target table corresponding to the data source table is generated in the target instance library. According to the embodiment described above, the synchronization tool synchronizes the user table of library A in the source instance mysql to the user table of library A in the hive instance. Specifically, the synchronization tool first creates a user table in library A of the hive instance. The created user table has the same form as the data source table in the source instance. The created user table is the target table in this step.

[0068] In this specification, the type of target table created in the target instance library varies according to the different synchronization tasks, and this application does not limit this.

[0069] Step 202: The business data in the data source table is transferred to the target table. After the target table is generated in step 201, the synchronization tool synchronizes the business data.

[0070] According to the previous embodiment, the data in the data source table in the source instance is the user table of library A in MySQL. The synchronization tool synchronizes the business data in the user table of library A in MySQL to the user table created in library A of the hive instance. As a result, the user table in library A of the hive instance currently obtains the synchronized business data.

[0071] Figure 3 The above is a flow chart of a method for updating basic metadata according to an embodiment of this specification, which specifically includes the following steps:

[0072] Step 301 : According to a pre-period, basic metadata in the source instance and the target instance are collected regularly, and the basic metadata is used to mark the information of each storage unit in each instance.

[0073] In the embodiments of this specification, the basic metadata of each instance may change. For example, two tables are added to the B library in the MySQL instance on T+1; for another example, several fields are added to a table in the C library in the Doris instance on T+1. Therefore, after completing the synchronization of business data, the basic metadata in the source instance and the target instance are regularly collected to achieve optimized management of data synchronization. Among them, the basic metadata is used to mark and maintain the information of each specific storage unit in each instance, such as table name, table field, etc.

[0074] Step 302, compare the basic metadata collected in the last two times to determine whether the basic metadata of each instance has changed. In this step, metadata change information is extracted by comparing the basic metadata information collected in the last two times. Among them, metadata change information includes but is not limited to: table deletion, table addition, table data storage location change, table remark change, table addition and deletion of fields, table field type change, table field annotation change, etc. By comparing the basic metadata of the last two times, it is possible to collect historical change information of each instance during the development process and improve data traceability.

[0075] Step 303: If yes, update the basic metadata of each instance.

[0076] In this step, if it is determined that the basic metadata of a certain instance collected twice most recently has changed, the basic metadata of the instance is updated with the basic metadata collected most recently.

[0077] Figure 4 The flowchart of a method for querying detailed information of a target table according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0078] Step 401: Determine the data source table and the target table according to the mapping relationship between the source instance and the target instance in the synchronization tool. Figure 2 In the illustrated embodiment, the synchronization tool determines the mapping relationship between the source instance and the target instance during the process of synchronizing business data from the source instance to the target instance. In other words, the tool can parse the mapping relationship between the source instance and the target instance, and further determine the data source table and the target table corresponding to the synchronization task.

[0079] Step 402, according to the table name of the target table, query the detailed information of the target table from the basic metadata of the data source table. In this step, the synchronization tool obtains the table name of the target table after generating the target table in the target instance library during the data synchronization process. Further, according to the table name, the basic metadata of the data source table is searched, and the specific information of the data source table is queried from the basic metadata of the data source table. The target table in the target instance library has a mapping relationship with the data source table, and the table information and table data of the two types of tables are the same, so the basic metadata of the data source table queried can be used as the basic metadata of the target table. The basic metadata of the data source is used as the detailed information of the target table.

[0080] Figure 5The flowchart of a method for establishing a bloodline link in an embodiment of this specification is shown. In this embodiment of this specification, the source instance and the target instance are deployed in the same business department or in different business departments. Specifically, the business demand for synchronizing business data can be synchronizing data in the same business department, or it can be synchronizing data in different business departments. For example, the databases of the business department include MySQL and Kafka, and the data warehouses of the big data department include Hive data warehouses and Doris data warehouses. Figure 5 The method specifically comprises the following steps:

[0081] Step 501 : According to the basic information of the first target instance recorded in the synchronization configuration information, detailed information of the first target table in the first target instance is matched from the basic metadata of the source instance.

[0082] In some embodiments of the present specification, the synchronization configuration information further includes, in addition to the basic information of the target table and the mapping relationship between the source instance and the target instance, basic information of the first target table configured in the first target instance, and the mapping relationship between the source instance, the target instance and the first target instance.

[0083] It can be understood that a synchronization task includes data synchronization between multiple instances, and the first target table is a target table in other instances other than the target table. The first target table can be an intermediate instance passed through in the process of synchronizing business data from the source instance to the target instance. For example, the source instance is mysql, the target instance is the topic in kafka, and the first target instance is the hive data warehouse. The synchronization configuration information includes the basic information of the hive data warehouse, the basic information of the kafka topic (such as the table name, etc.), and the mapping relationship between the source instance and the first target instance and the target instance.

[0084] Step 502: Establish a lineage link between the business end and the target instance based on the mapping relationship, the basic metadata of the source instance, the detailed information of the target table in the target instance, and the detailed information of the first target table of the first target instance. Similar to step 104, establish a lineage link between the business end and the target instance.

[0085] like Figure 6 The figure shows a schematic diagram of the structure of a bloodline data management device based on a data warehouse link in an embodiment of the present specification. The figure describes the basic structure of the bloodline data management device based on a data warehouse link. The functional units and modules therein can implement bloodline data management based on a data warehouse link in a software manner, or can be implemented using a general chip or a specific chip. The device specifically includes:

[0086] The synchronization unit 601 is used to establish a synchronization task of business data according to business requirements, and synchronize business data from a source instance to a target instance using a synchronization tool.

[0087] The collection unit 602 is used to collect the basic metadata of the source instance and the synchronization configuration information of the synchronization tool respectively. The synchronization configuration information records the basic information of the target table configured in the target instance by the synchronization tool during the business data synchronization process and the mapping relationship between the source instance and the target instance.

[0088] The matching unit 603 is used to match detailed information of the target table from the basic metadata of the source instance according to the basic information.

[0089] The lineage link establishing unit 604 is used to establish a lineage link between the service end and the target instance according to the mapping relationship, the basic metadata of the source instance and the detailed information of the target table.

[0090] This solution aims to provide efficient metadata management and better manage and maintain metadata, including data definitions, attributes, patterns, relationships, etc. It provides metadata visualization, centralized management and fast retrieval functions.

[0091] Provides a scalable and flexible system architecture to adapt to the changing and growing data governance needs of the organization. Able to integrate with existing data management tools and systems, providing open interfaces and standards to facilitate system deployment and expansion. Provides a scalable and flexible architecture.

[0092] As an example of this specification, please refer to Figure 7 The figure shows a specific structural diagram of a bloodline data management device based on a data warehouse link according to an embodiment of the present specification.

[0093] As an embodiment of this specification, the acquisition unit 602 further includes:

[0094] Comparison module 6021, used to compare the two most recently collected basic metadata to determine whether the basic metadata of each instance has changed;

[0095] The update module 6022 is used to update the basic metadata of each instance if yes.

[0096] like Figure 8As shown, it is a schematic diagram of a computer device provided by an embodiment of this specification. The bloodline data management method based on the data warehouse link described in this specification can be executed by a computer device. The computer device 802 may include one or more processors 804, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 802 may also include any memory 806, which is used to store any kind of information such as code, settings, data, etc. Non-limiting, for example, the memory 806 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 802. In one case, when the processor 804 executes an associated instruction stored in any memory or combination of memories, the computer device 802 can perform any operation of the associated instruction. The computer device 802 also includes one or more drive mechanisms 808 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0097] The computer device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via input devices 812) and for providing various outputs (via output devices 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface (GUI) 818. In other embodiments, the input / output module 810 (I / O), the input device 812, and the output device 814 may not be included, and the computer device 802 may be used as a computer device in a network. The computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0098] The communication link 822 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0099] Corresponds to Figures 1 to 5 The method in the embodiment of the present specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0100] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 5 The method shown.

[0101] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0102] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0103] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0107] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0109] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A bloodline data management method based on data warehouse link, characterized in that: The method comprises: Establish business data synchronization tasks according to business needs, and use synchronization tools to synchronize business data from the source instance to the target instance; The basic metadata of the source instance and the synchronization configuration information of the synchronization tool are collected respectively, wherein the synchronization configuration information records the basic information of the target table configured in the target instance by the synchronization tool during the business data synchronization process and the mapping relationship between the source instance and the target instance; According to the basic information, matching detailed information of the target table from the basic metadata of the source instance; A lineage link between the business end and the target instance is established based on the mapping relationship, the basic metadata of the source instance and the detailed information of the target table.

2. The bloodline data management method based on data warehouse link according to claim 1 is characterized in that: Using the synchronization tool to synchronize business data from the source instance to the target instance includes: Generate a target table in the target instance library according to the target instance library in the synchronization task and the data source table in the source instance, wherein the target table corresponds to the data source table; The business data in the data source table is transferred to the target table.

3. The bloodline data management method based on data warehouse link according to claim 2 is characterized in that: The basic metadata collected for each instance also includes: According to the pre-period, basic metadata in the source instance and the target instance are collected regularly, and the basic metadata is used to annotate the information of each storage unit in each instance; Compare the two most recently collected basic metadata to determine whether the basic metadata of each instance has changed; If yes, update the basic metadata of each instance.

4. The bloodline data management method based on data warehouse link according to claim 3 is characterized in that: According to the basic information, detailed information of the target table is matched from the basic metadata of the source instance, including: Determine the data source table and target table based on the mapping relationship between the source instance and the target instance in the synchronization tool; According to the table name of the target table, detailed information of the target table is queried from the basic metadata of the data source table.

5. The bloodline data management method based on data warehouse link according to claim 4 is characterized in that: The synchronization configuration information further includes: basic information of the first target table configured in the first target instance, and a mapping relationship between the source instance, the target instance, and the first target instance. According to the mapping relationship, the basic metadata of the source instance, and the detailed information of the target table, establishing a lineage link between the service end and the target instance further includes: According to the basic information of the first target instance recorded in the synchronization configuration information, matching detailed information of the first target table in the first target instance from the basic metadata of the source instance; A lineage link between the business end and the target instance is established based on the mapping relationship, the basic metadata of the source instance, the detailed information of the target table in the target instance, and the detailed information of the first target table of the first target instance.

6. The bloodline data management method based on data warehouse link according to claim 1 is characterized in that: The source instance and the target instance are deployed in the same business department or in different business departments.

7. The bloodline data management method based on data warehouse link according to claim 6 is characterized in that: If an abnormality occurs in the data in the bloodline link, the cause of the abnormality is determined based on the basic metadata of the abnormal data.

8. A bloodline data management device based on a data warehouse link, characterized in that: The device comprises: The synchronization unit is used to establish synchronization tasks for business data according to business needs and use synchronization tools to synchronize business data from the source instance to the target instance; A collection unit, used to collect basic metadata of the source instance and synchronization configuration information of the synchronization tool, respectively, wherein the synchronization configuration information records basic information of a target table configured in a target instance by the synchronization tool during synchronization of business data and a mapping relationship between a source instance and a target instance; A matching unit, configured to match detailed information of a target table from basic metadata of a source instance according to the basic information; A lineage link establishing unit is used to establish a lineage link between the business end and the target instance according to the mapping relationship, the basic metadata of the source instance and the detailed information of the target table.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.