Data blood relationship processing method and device and storage medium

Abstract

The invention discloses a data blood relationship processing method and device and a storage medium. The method comprises the steps that according to a preset collection assembly, out-of-warehouse data synchronization information of a plurality of out-of-warehouse databases corresponding to an in-warehouse database is obtained, the in-warehouse database is a database inside a data warehouse, and the out-of-warehouse databases are databases of a business system outside the data warehouse; the out-of-warehouse data synchronization information is used for representing data synchronization information between different out-of-warehouse databases; analyzing the out-of-warehouse data synchronization information to obtain out-of-warehouse blood relationship metadata; a data consanguinity knowledge graph is generated according to the out-of-warehouse consanguinity metadata and the in-warehouse database, and the data consanguinity knowledge graph is used for representing the data consanguinity between the in-warehouse database and the multiple out-of-warehouse databases. Through the method and the device, the problem that the coverage range of the data blood relationship is relatively small in the related technology is solved.

Description

Technical Field

[0001] This application relates to the field of data management, and more specifically, to a method, apparatus, and storage medium for processing data lineage. Background Technology

[0002] Data lineage management in data warehouses is one of the important tasks of data management. In traditional data lineage management, the focus is mainly on the data processing flow within the warehouse, such as tracing the dependencies between tables or fields generated by the execution of database statements.

[0003] However, realizing the value of data requires a complete process: data generation by business systems, cross-system synchronization, and in-warehouse processing and analysis. The data synchronization logic between external business systems has long been a blind spot in data lineage management. This results in the data flow from the business source to data warehouse applications being untraceable. When business systems iterate or data warehouse logic is adjusted, it is difficult to accurately pinpoint the scope of impact. Data governance can only passively fix issues within the warehouse, and data lineage relationships are difficult to cover the business and data warehouse development sides, hindering the realization of deep cross-system data collaboration needs.

[0004] There is currently no effective solution to the problem of the limited coverage of data lineage in related technologies. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, and storage medium for processing data lineage relationships, so as to solve the problem of the limited coverage of data lineage relationships in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a method for processing data lineage is provided. The method includes: acquiring external data synchronization information of multiple external databases corresponding to an internal database using a preset acquisition component, wherein the internal database is a database within the data warehouse, and the external databases are databases of business systems outside the data warehouse; the external data synchronization information is used to represent data synchronization information between different external databases; parsing the external data synchronization information to obtain external lineage metadata; and generating a data lineage knowledge graph based on the external lineage metadata and the internal database, wherein the data lineage knowledge graph is used to represent the data lineage relationships between the internal database and multiple external databases.

[0007] Furthermore, obtaining external data synchronization information for multiple external databases corresponding to the internal database based on preset acquisition components includes: obtaining system architecture information of multiple business systems and determining data interaction nodes corresponding to multiple external databases based on the system architecture information; installing preset acquisition components on the data interaction nodes and collecting data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies corresponding to the data interaction nodes based on the preset acquisition components; and determining external data synchronization information based on data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies.

[0008] Furthermore, the external data synchronization information is parsed to obtain external lineage metadata, including: parsing the external data synchronization information using preset syntax rules to obtain syntax parsing results; verifying the syntax parsing results according to preset business rules to obtain verification results; and determining the external lineage metadata based on the syntax parsing results if the verification result is successful.

[0009] Furthermore, generating a data lineage knowledge graph based on external lineage metadata and internal database includes: associating external lineage metadata with internal lineage metadata in the internal database to obtain metadata association results; determining the data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies of multiple business systems; and generating a data lineage knowledge graph based on the metadata association results, business system data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies.

[0010] Furthermore, after generating a data lineage knowledge graph based on external lineage metadata and internal database, the method further includes: monitoring interface change information of multiple business systems to obtain first change information; obtaining data warehouse change logs from the internal database and determining second change information based on the data warehouse change logs; determining target change information based on the first and second change information and determining lineage change information based on the association results of the target change information and metadata; and updating the data lineage knowledge graph based on the lineage change information.

[0011] Furthermore, after generating a data lineage knowledge graph based on external lineage metadata and internal database, the method further includes: receiving requirement change information for the business system; performing deduction in the data lineage knowledge graph based on the requirement change information to obtain the requirement deduction result; and determining the data warehouse modification prediction result corresponding to the requirement change information based on the requirement deduction result, wherein the data warehouse modification prediction result is used to represent the database modification operation to be performed by the internal database in response to the requirement change information.

[0012] To achieve the above objectives, according to another aspect of this application, a data lineage processing apparatus is provided. The apparatus includes: an information acquisition unit, configured to acquire external data synchronization information of multiple external databases corresponding to an internal database based on a preset acquisition component, wherein the internal databases are databases within the data warehouse, and the external databases are databases of business systems outside the data warehouse, and the external data synchronization information represents data synchronization information between different external databases; an information parsing unit, configured to parse the external data synchronization information to obtain external lineage metadata; and a graph generation unit, configured to generate a data lineage knowledge graph based on the external lineage metadata and the internal databases, wherein the data lineage knowledge graph represents the data lineage relationships between the internal databases and multiple external databases.

[0013] Furthermore, the information acquisition unit includes: an interaction node determination module, used to acquire system architecture information of multiple business systems and determine data interaction nodes corresponding to multiple off-site databases based on the system architecture information; a data acquisition module, used to install preset acquisition components on the data interaction nodes and acquire data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies corresponding to the data interaction nodes based on the preset acquisition components; and a synchronization information determination module, used to determine off-site data synchronization information based on data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies.

[0014] Furthermore, the information parsing unit includes: a syntax parsing module, used to parse the off-warehouse data synchronization information using preset syntax rules to obtain syntax parsing results; a rule verification module, used to verify the syntax parsing results according to preset business rules to obtain verification results; and a metadata determination module, used to determine off-warehouse lineage metadata based on the syntax parsing results if the verification result is successful.

[0015] Furthermore, the graph generation unit includes: a metadata association module, used to associate external lineage metadata with internal lineage metadata in the internal database to obtain metadata association results; an entity relationship determination module, used to determine the data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies of multiple business systems; and a generation module, used to generate a data lineage knowledge graph based on the metadata association results, data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies.

[0016] Furthermore, the device also includes: a first change unit, used to monitor interface change information of multiple business systems and obtain first change information after generating a data lineage knowledge graph based on external lineage metadata and internal database; a second change module, used to obtain data warehouse change logs from internal database and determine second change information based on the data warehouse change logs; a lineage change module, used to determine target change information based on the first and second change information and determine lineage change information based on the target change information and metadata association results; and a graph update module, used to update the data lineage knowledge graph based on the lineage change information.

[0017] Furthermore, the device also includes: a demand receiving unit, used to receive demand change information for the business system after generating a data lineage knowledge graph based on external lineage metadata and internal database; a graph derivation unit, used to deduce the demand from the data lineage knowledge graph based on the demand change information to obtain the demand derivation result; and a modification prediction unit, used to determine the data warehouse modification prediction result corresponding to the demand change information based on the demand derivation result, wherein the data warehouse modification prediction result is used to represent the database modification operation to be performed by the internal database in response to the demand change information.

[0018] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any data lineage processing method.

[0019] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a processing method for performing any kind of data lineage.

[0020] According to another aspect of this application, a computer program product is provided, including computer instructions, wherein when the computer instructions are executed by a processor, the steps of a method for processing data lineage relationships as described above are implemented.

[0021] In this embodiment, external data synchronization information of multiple external databases corresponding to the internal database is obtained by using a preset acquisition component. The internal database is the database inside the data warehouse, and the external database is the database of the business system outside the data warehouse. The external data synchronization information is used to represent the data synchronization information between different external databases. The external data synchronization information is parsed to obtain external lineage metadata. A data lineage knowledge graph is generated based on the external lineage metadata and the internal database. The data lineage knowledge graph is used to represent the data lineage relationship between the internal database and multiple external databases, which solves the technical problem of the small coverage of data lineage relationship in the prior art. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data lineage processing method is shown.

[0024] Figure 2 This is a flowchart of a data lineage processing method provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram illustrating the generation of a data lineage knowledge graph based on the data lineage processing method provided in the embodiments of this application;

[0026] Figure 4 This is a schematic diagram of a data lineage processing apparatus provided according to an embodiment of this application;

[0027] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0031] Example 1

[0032] According to an embodiment of this application, a method embodiment for processing data lineage is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a data lineage processing method is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data lineage processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned data lineage processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0037] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0038] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for handling blood relations in the data is shown. Figure 2 This is a flowchart of a method for processing data lineage according to Embodiment 1 of this application.

[0039] Step S201: Obtain the external data synchronization information of multiple external databases corresponding to the internal database according to the preset acquisition components.

[0040] It should be noted that the internal database refers to the database within the data warehouse, while the external database refers to the databases of business systems outside the data warehouse. External data synchronization information is used to represent data synchronization information between different external databases, and the preset collection component can be a lightweight collection component. For example, data synchronization information could be the data synchronization information between the database of business system A and the database of business system B, but this synchronization information is not recorded in the internal database.

[0041] Optionally, the aforementioned data warehouse can be a big data warehouse. In existing technologies, data lineage is established solely based on the internal database and stored within the big data warehouse, making it impossible to link to data lineages in external databases. Depending on the project, different external business systems may exist for the data warehouse. For example, these business systems could be a visual connectivity operation management platform, an intelligent network management platform, or a business support system.

[0042] Step S202: Parse the data synchronization information outside the warehouse to obtain the external bloodline metadata.

[0043] Optionally, off-site data synchronization information is usually unstructured, such as ETL (Extract, Transform, Load, etc.) script fragments or program interface interaction parameters. It is difficult to directly generate corresponding metadata based on off-site data synchronization information. The off-site data synchronization information can be parsed first to convert the unstructured information into structured information, and then off-site lineage metadata can be generated based on the structured information.

[0044] Step S203: Generate a data lineage knowledge graph based on external lineage metadata and internal database.

[0045] Optionally, an open-source graph database can be chosen as the storage engine for generating the data lineage knowledge graph. Entities in the data lineage knowledge graph can be defined, including business system data source entities, data warehouse table entities, etc. Relationships between entities can include synchronization channel relationships, synchronization dependencies, etc., thus determining the schema (i.e., conceptual layer) of the data lineage knowledge graph based on entities and relationships. Internal data lineage metadata can be determined based on the internal database, and external data lineage metadata can be associated with internal data lineage metadata to obtain metadata association results. These metadata association results represent the full-cycle data lineage metadata under the condition of internal and external data fusion. Based on the above metadata association results (i.e., the fused metadata), a graph construction script or a graphical interface tool can be written to map entities and relationships to the graph database, forming a data lineage knowledge graph that includes the full-cycle data lineage relationships of external synchronization and internal processing. In other words, the data lineage knowledge graph represents the data lineage relationships between the internal database and multiple external databases.

[0046] In summary, by acquiring the external data synchronization information from the external database and parsing it to obtain structured external lineage metadata, a data lineage knowledge graph containing the data lineage relationships throughout the entire cycle of external synchronization and internal processing can be generated based on the external lineage metadata and the internal database, thereby increasing the coverage of data lineage relationships.

[0047] To broaden the coverage of data lineage, optionally, acquiring external data synchronization information of multiple external databases corresponding to the internal database using preset acquisition components includes: acquiring system architecture information of multiple business systems and determining data interaction nodes corresponding to multiple external databases based on the system architecture information; installing preset acquisition components on the data interaction nodes and acquiring data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies corresponding to the data interaction nodes based on the preset acquisition components; and determining external data synchronization information based on data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies.

[0048] Optionally, system architecture information can be obtained from multiple business systems. This information may include the module divisions, data flow diagrams, interface documents, and data synchronization strategy documents of the business systems, representing the flow path of off-warehouse data between the databases of different business systems. Based on the system architecture information, data interaction nodes for data interaction between the databases of each business system can be determined, such as program interface nodes, database replication channels, and entry points for ETL jobs. Pre-set acquisition components can be installed on the data interaction nodes. These components monitor and record key information during the data interaction process, such as data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies. Data field mapping relationships record the conversion rules between data source fields and target fields; data format conversion protocols represent the format changes during data transmission; data synchronization conditions record the triggering conditions for data synchronization (e.g., filtering rules, synchronization thresholds); and data synchronization frequencies record the data synchronization cycle. Integrating the data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies yields off-warehouse data synchronization information.

[0049] In summary, by acquiring system architecture information from multiple business systems and determining data interaction nodes corresponding to multiple off-warehouse databases based on this information, and by installing pre-defined data acquisition components at these nodes, the system collects data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies corresponding to each node. Based on these data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies, off-warehouse data synchronization information is determined, laying the data foundation for acquiring off-warehouse lineage metadata and thus increasing the coverage of data lineage relationships.

[0050] To broaden the coverage of data lineage, optionally, the data synchronization information outside the warehouse is parsed to obtain the data lineage metadata outside the warehouse, including: parsing the data synchronization information outside the warehouse using preset syntax rules to obtain the syntax parsing result; verifying the syntax parsing result according to preset business rules to obtain the verification result; and determining the data lineage metadata outside the warehouse based on the syntax parsing result if the verification result is successful.

[0051] Optionally, Figure 3 This is a schematic diagram illustrating the generation of a data lineage knowledge graph based on the data lineage processing method provided in the embodiments of this application. (Refer to...) Figure 3As shown, preprocessing of external data synchronization information can be performed to remove redundant information such as script comments and program interface control fields. A unified encoding format of UTF-8 (one encoding format) can be used to ensure parsing accuracy. Preset syntax rules can be read from the business rule library. These preset business rules can be stored in JSON format and may include data synchronization rules for the business system, field mapping standards and data format requirements, and conversion rules for different synchronization scenarios. For example, in the synchronization scenario of the VisionLink Operation Management Platform - Intelligent Network Management Platform, the resource type (level 1) field is automatically mapped to the resource type field of the Intelligent Network Management Platform, triggering a conversion rule to remove spaces from field values. Verifying the syntax parsing results based on the preset business rules involves checking the correctness of field mapping, the compliance of data format conversion, and the reasonableness of synchronization conditions and frequency. Determining external lineage metadata based on syntax parsing results can be achieved by using the parsing results as the external lineage metadata, or by integrating the parsing results according to the rules of data source, target table, field mapping, and synchronization conditions to generate structured external lineage metadata. For example, external lineage metadata could include: Business system data source: a business system customer database; Target table: the customer table in the data warehouse ODS layer (i.e., the operational data storage layer); Field mapping: customer ID, customer ID, customer name, customer name; Synchronization condition: customer status is valid.

[0052] In summary, by using preset syntax rules to parse the data synchronization information outside the warehouse, a syntax parsing result is obtained; the syntax parsing result is verified according to preset business rules, and a verification result is obtained; if the verification result is successful, the data lineage metadata outside the warehouse is determined based on the syntax parsing result, thereby increasing the coverage of data lineage relationships.

[0053] To broaden the coverage of data lineage relationships, optionally, generating a data lineage knowledge graph based on external lineage metadata and internal databases includes: associating external lineage metadata with internal lineage metadata in the internal database to obtain metadata association results; determining business system data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies based on multiple business systems; and generating a data lineage knowledge graph based on the metadata association results, business system data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies.

[0054] Optionally, an open-source graph database can be selected to create basic indexes (e.g., indexing business data source nodes by unique codes, and indexing data warehouse table nodes by table IDs) to improve the efficiency of cross-domain lineage queries. (See reference) Figure 3As shown, external lineage metadata can be associated with internal lineage metadata in the internal database. Specifically, the unique code of the business system's data source and the interface version number can be associated with the corresponding data source identifier field in the internal database to establish the starting point association of the data flow. The synchronization channel ID can be associated with the task ID that the data warehouse scheduling depends on to track the specific execution path of data synchronization from external to internal to the data warehouse. By associating the data warehouse table ID with the target table identifier in the database execution log, the process of how the database processes data from one table to another can be tracked. Based on multiple business systems, the data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies of the business systems are determined. This can involve defining data source entities (e.g., entities for the VisionLink Operations Management Platform and the Intelligent Network Management Platform, whose attributes include system unique code, interface version number, and data format) and data warehouse table entities (i.e., table entities in the data warehouse, such as tables in the ODS layer, DWD layer, and DWS layer, whose attributes include table level, table structure, and business theme). The ODS layer is the operational data storage layer, the DWD layer is the data warehouse detail layer, and the DWS layer is the data warehouse summary layer. It should be noted that the number of data source entities and data warehouse table entities can be multiple, depending on the business system. Synchronization channel relationships and synchronization dependencies are defined for the knowledge graph concept layer. Synchronization channel relationships can include synchronization channels such as the ETL channel between the VisionLink Operations Management Platform and the Intelligent Network Management Platform, while synchronization dependencies represent the dependency links for synchronizing data between business systems. In addition to the relationships mentioned above, processing dependencies and field derivation relationships can be defined for the conceptual layer of the knowledge graph. Processing dependencies refer to the dependencies of in-warehouse table processing on upstream data; for example, the processing dependency of a customer detail table at a certain layer of the data warehouse on a customer table at another layer. Field derivation relationships refer to the transformation and generation relationships of fields during data flow; for example, the transformation and generation relationship between a customer's mobile phone number in a business system and the mobile phone number prefix at a certain layer of the data warehouse. Scripts can be used to generate business system data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies in the graph database. Based on the metadata association results read by the scripts, corresponding nodes and relationships are created in the graph database to obtain the data lineage knowledge graph. The data lineage knowledge graph displays the entire chain of customer fields in the business system, customer tables in the data warehouse ODS layer, processing dependencies, customer detail tables in the data warehouse DWD layer, processing dependencies, and customer value indicator tables in the data warehouse DWS layer. It supports cross-domain lineage association queries (e.g., querying the impact range of changes in customer tag fields in a business system on customer value indicators in the data warehouse DWS layer).

[0055] Optionally, after generating the data lineage knowledge graph, it can receive query requests from business personnel and traverse the data lineage knowledge graph according to the query requests to obtain the complete path from the business side (i.e., the business system) to the data warehouse application side (i.e., the database within the warehouse), which makes it easier for business personnel to understand the data flow logic throughout the entire lifecycle.

[0056] In summary, by associating external lineage metadata with internal lineage metadata in the internal database, metadata association results are obtained; based on multiple business systems, the data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies of the business systems are determined; and based on the metadata association results, data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies, a data lineage knowledge graph is generated, thereby increasing the coverage of data lineage relationships.

[0057] To broaden the coverage of data lineage relationships, optionally, after generating a data lineage knowledge graph based on external lineage metadata and internal database, the method further includes: monitoring interface change information of multiple business systems to obtain first change information; obtaining data warehouse change logs from the internal database and determining second change information based on the data warehouse change logs; determining target change information based on the first and second change information, and determining lineage change information based on the association results of the target change information and metadata; and updating the data lineage knowledge graph based on the lineage change information.

[0058] Optionally, interface change information for each business system can be monitored using polling or time-driven methods to obtain the first change information. Database statement script changes (e.g., modifications to field mapping logic in the data warehouse ODS and DWD layers) can be monitored using data warehouse scheduling tool logs. Changes such as field additions, deletions, or type modifications in data warehouse tables can be identified through data warehouse table structure change logs. Integrating these database statement script changes and data warehouse table field additions, deletions, or type modifications yields the second change information. The first and second change information are then combined to obtain the target change information. The target change information is compared with the metadata association results to determine lineage change points (e.g., if a business system adds a customer tag field, it is identified as a field mapping addition change point in data warehouse synchronization for that business system; if the data warehouse database statement script modifies the conversion rules for customer mobile phone numbers or mobile phone number prefixes, it is identified as a field derivation relationship modification change point in the ODS layer). All lineage change points are identified as lineage change information. The data lineage knowledge graph is then updated based on this information to obtain the updated data lineage knowledge graph. Before updating the data lineage knowledge graph, a version snapshot of the data lineage knowledge graph can be created for easy version rollback.

[0059] In summary, by monitoring interface change information of multiple business systems, the first change information is obtained; the data warehouse change log of the internal database is obtained, and the second change information is determined based on the data warehouse change log; the target change information is determined based on the first and second change information, and the lineage change information is determined based on the target change information and the metadata association results; the data lineage knowledge graph is updated based on the lineage change information, thereby increasing the coverage of data lineage relationships.

[0060] To improve the efficiency of data management based on data lineage, the method may optionally include, after generating a data lineage knowledge graph based on external lineage metadata and internal database, receiving requirement change information for the business system; performing deduction in the data lineage knowledge graph based on the requirement change information to obtain the requirement deduction result; and determining the data warehouse modification prediction result corresponding to the requirement change information based on the requirement deduction result, wherein the data warehouse modification prediction result is used to represent the database modification operation to be performed by the internal database in response to the requirement change information.

[0061] Optionally, when the business department removes requirement change information for the business system, such as adding data fields or modifying data processing logic, it can receive the aforementioned requirement change information and use graph database query language to deduce path traversal in the data lineage knowledge graph to determine the entities and relationships affected by the requirement change information in the data lineage knowledge graph. The entities and relationships affected by the requirement change information are identified as the requirement deduction results. The requirement deduction results characterize the scope of the impact of the requirement change information on the data lineage relationship. Based on the requirement deduction results, the database modification operations to be performed in response to the requirement change information are determined, such as the modification operations to be performed on the warehouse table name, processing logic, etc., which helps personnel shorten the requirement implementation cycle.

[0062] Optionally, when an anomaly is detected in the data warehouse metrics (e.g., inconsistent customer value metrics in the DWS layer), a reverse tracing can be initiated by calling the query interface of the data lineage knowledge graph. This allows for starting from the DWS layer tables associated with the anomaly, traversing along processing dependencies to the DWD layer tables, then the ODS layer tables, and finally along synchronization dependencies to the external business system data source, thus identifying problems at each stage. For example, if the ODS layer table data is inconsistent with the original data in the business system, the problem is identified as a defect in the synchronization transformation rules; if the original data in the business system contains null values, the problem is identified as an error in the original data. A tracing report containing problem location, error type, and troubleshooting suggestions can be output to improve the efficiency of anomaly repair in the business system.

[0063] In summary, by receiving change requests for business systems and deriving the changes from these requests in a data lineage knowledge graph, the system simulates changes based on these changes within the data lineage knowledge graph, which includes data lineage relationships from the business side. This allows the system to obtain the database modification operations that the in-warehouse database needs to perform in response to the change requests, shortening the requirement implementation cycle and improving the efficiency of data management based on data lineage relationships.

[0064] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0065] Example 2

[0066] This application also provides a data lineage processing apparatus. It should be noted that this data lineage processing apparatus can be used to execute the data lineage processing method provided in this application. The following describes the data lineage processing apparatus provided in this application.

[0067] According to embodiments of this application, an apparatus for implementing the above-described data lineage processing method is also provided, such as... Figure 4 As shown, the device includes:

[0068] The information acquisition unit 401 is used to acquire the external data synchronization information of multiple external databases corresponding to the internal database according to the preset acquisition components. The internal database is the database inside the data warehouse, and the external database is the database of the business system outside the data warehouse. The external data synchronization information is used to represent the data synchronization information between different external databases.

[0069] The information parsing unit 402 is used to parse the data synchronization information outside the warehouse to obtain the external lineage metadata.

[0070] The graph generation unit 403 is used to generate a data lineage knowledge graph based on external lineage metadata and internal database. The data lineage knowledge graph is used to represent the data lineage relationships between the internal database and multiple external databases.

[0071] The data lineage processing apparatus provided in this application embodiment acquires external data synchronization information of multiple external databases corresponding to the internal database using an information acquisition unit 401 based on a preset acquisition component. The internal databases are databases within the data warehouse, while the external databases are databases of business systems outside the data warehouse. The external data synchronization information represents the data synchronization information between different external databases. The information parsing unit 402 parses the external data synchronization information to obtain external lineage metadata. The graph generation unit 403 generates a data lineage knowledge graph based on the external lineage metadata and the internal databases. This data lineage knowledge graph represents the data lineage relationships between the internal databases and multiple external databases, solving the problem of limited coverage of data lineage relationships in related technologies. This effectively increases the coverage of data lineage relationships.

[0072] Optionally, in the data lineage processing apparatus provided in this application embodiment, the information acquisition unit 401 includes: an interaction node determination module, used to acquire system architecture information of multiple business systems and determine data interaction nodes corresponding to multiple off-site databases based on the system architecture information; a data acquisition module, used to install preset acquisition components on the data interaction nodes and acquire data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies corresponding to the data interaction nodes based on the preset acquisition components; and a synchronization information determination module, used to determine off-site data synchronization information based on data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies.

[0073] Optionally, in the data lineage processing apparatus provided in this application embodiment, the information parsing unit 402 includes: a syntax parsing module, used to parse the data synchronization information outside the warehouse using preset syntax rules to obtain a syntax parsing result; a rule verification module, used to verify the syntax parsing result according to preset business rules to obtain a verification result; and a metadata determination module, used to determine the outside lineage metadata based on the syntax parsing result if the verification result is successful.

[0074] Optionally, in the data lineage processing apparatus provided in this application embodiment, the graph generation unit 403 includes: a metadata association module, used to associate external lineage metadata with internal lineage metadata in the internal database to obtain a metadata association result; an entity relationship determination module, used to determine the data source entity of the business system, the data warehouse table entity, the synchronization channel relationship, and the synchronization dependency relationship based on multiple business systems; and a generation module, used to generate a data lineage knowledge graph based on the metadata association result, the data source entity of the business system, the data warehouse table entity, the synchronization channel relationship, and the synchronization dependency relationship.

[0075] Optionally, in the data lineage processing apparatus provided in this application embodiment, the apparatus further includes: a first change unit, used to monitor interface change information of multiple business systems and obtain first change information after generating a data lineage knowledge graph based on external lineage metadata and internal database; a second change module, used to obtain data warehouse change logs of internal database and determine second change information based on data warehouse change logs; a lineage change module, used to determine target change information based on the first change information and second change information, and determine lineage change information based on the target change information and metadata association results; and a graph update module, used to update the data lineage knowledge graph based on the lineage change information.

[0076] Optionally, in the data lineage processing apparatus provided in this application embodiment, the apparatus further includes: a demand receiving unit, configured to receive demand change information for the business system after generating a data lineage knowledge graph based on external lineage metadata and internal database; a graph derivation unit, configured to deduce the demand in the data lineage knowledge graph based on the demand change information to obtain a demand derivation result; and a modification prediction unit, configured to determine the data warehouse modification prediction result corresponding to the demand change information based on the demand derivation result, wherein the data warehouse modification prediction result is used to represent the database modification operation to be performed by the internal database in response to the demand change information.

[0077] It should be noted that the information acquisition unit 401, information parsing unit 402, and map generation unit 403 mentioned above correspond to steps S201 to S203 in Embodiment 1. The instances and application scenarios implemented by the units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0078] Example 3

[0079] Embodiments of this application may provide an electronic device. Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (Only one is shown) processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0080] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0081] The processor can access the information and application programs stored in the memory via a transmission device to execute the following steps: acquiring external data synchronization information of multiple external databases corresponding to the internal database based on a preset acquisition component. The internal database refers to the database within the data warehouse, while the external databases are databases of business systems outside the data warehouse. The external data synchronization information represents the data synchronization information between different external databases. The external data synchronization information is then parsed to obtain external lineage metadata. A data lineage knowledge graph is generated based on the external lineage metadata and the internal databases. This data lineage knowledge graph represents the data lineage relationships between the internal databases and multiple external databases.

[0082] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain system architecture information of multiple business systems, and determine the data interaction nodes corresponding to multiple off-site databases based on the system architecture information; install preset acquisition components on the data interaction nodes, and collect the data field mapping relationship, data format conversion protocol, data synchronization conditions and data synchronization frequency corresponding to the data interaction nodes based on the preset acquisition components; determine the off-site data synchronization information based on the data field mapping relationship, data format conversion protocol, data synchronization conditions and data synchronization frequency.

[0083] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: parse the data synchronization information outside the warehouse using preset syntax rules to obtain the syntax parsing result; verify the syntax parsing result according to preset business rules to obtain the verification result; and determine the off-warehouse lineage metadata based on the syntax parsing result if the verification result is successful.

[0084] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: associate the external lineage metadata with the internal lineage metadata in the internal database to obtain the metadata association result; determine the data source entities, data warehouse table entities, synchronization channel relationships and synchronization dependencies of multiple business systems; and generate a data lineage knowledge graph based on the metadata association result, the data source entities, data warehouse table entities, synchronization channel relationships and synchronization dependencies of the business systems.

[0085] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: listen for interface change information of multiple business systems to obtain first change information; obtain the data warehouse change log of the data warehouse database and determine the second change information based on the data warehouse change log; determine the target change information based on the first change information and the second change information, and determine the lineage change information based on the target change information and the metadata association result; update the data lineage knowledge graph based on the lineage change information.

[0086] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: receiving requirement change information for the business system; deriving the requirement derivation result in the data lineage knowledge graph based on the requirement change information; determining the data warehouse modification prediction result corresponding to the requirement change information based on the requirement derivation result, wherein the data warehouse modification prediction result is used to represent the database modification operation to be performed by the database in response to the requirement change information.

[0087] This application provides a solution for processing data lineage relationships. It acquires external data synchronization information from multiple external databases corresponding to an internal database using a preset acquisition component. The internal databases are databases within the data warehouse, while the external databases are databases of business systems outside the data warehouse. The external data synchronization information represents the data synchronization information between different external databases. The external data synchronization information is parsed to obtain external lineage metadata. A data lineage knowledge graph is generated based on the external lineage metadata and the internal databases. This data lineage knowledge graph represents the data lineage relationships between the internal databases and multiple external databases, solving the technical problem of limited coverage of data lineage relationships in existing technologies.

[0088] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0089] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0090] Example 4

[0091] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data lineage processing method provided in Embodiment 1.

[0092] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0093] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining external data synchronization information of multiple external databases corresponding to the internal database according to a preset acquisition component, wherein the internal database is a database inside the data warehouse, the external database is a database of a business system outside the data warehouse, and the external data synchronization information is used to represent the data synchronization information between different external databases; parsing the external data synchronization information to obtain external lineage metadata; generating a data lineage knowledge graph based on the external lineage metadata and the internal database, wherein the data lineage knowledge graph is used to represent the data lineage relationship between the internal database and multiple external databases.

[0094] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: obtaining system architecture information of multiple business systems, and determining data interaction nodes corresponding to multiple off-site databases based on the system architecture information; installing preset acquisition components on the data interaction nodes, and acquiring data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies corresponding to the data interaction nodes based on the preset acquisition components; and determining off-site data synchronization information based on the data field mapping relationships, data format conversion protocols, data synchronization conditions, and data synchronization frequencies.

[0095] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: parsing the off-site data synchronization information using preset syntax rules to obtain a syntax parsing result; verifying the syntax parsing result according to preset business rules to obtain a verification result; and determining the off-site lineage metadata based on the syntax parsing result if the verification result is successful.

[0096] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: associating external lineage metadata with internal lineage metadata in the internal database to obtain metadata association results; determining business system data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies based on multiple business systems; and generating a data lineage knowledge graph based on the metadata association results, business system data source entities, data warehouse table entities, synchronization channel relationships, and synchronization dependencies.

[0097] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: monitoring interface change information of multiple business systems to obtain first change information; obtaining data warehouse change logs of the in-warehouse database and determining second change information based on the data warehouse change logs; determining target change information based on the first change information and the second change information, and determining lineage change information based on the target change information and the metadata association result; and updating the data lineage knowledge graph based on the lineage change information.

[0098] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for performing the following steps: receiving requirement change information for the business system; performing inference in the data lineage knowledge graph based on the requirement change information to obtain the requirement inference result; determining the data warehouse modification prediction result corresponding to the requirement change information based on the requirement inference result, wherein the data warehouse modification prediction result is used to represent the database modification operation to be performed by the database in the warehouse in response to the requirement change information.

[0099] This application also provides a computer program product, which, when executed on a data processing device, is suitable for performing steps of a data lineage processing method.

[0100] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0101] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0106] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data lineage processing method, characterized by, The method comprises the following steps: obtaining warehouse-outside data synchronization information of a plurality of warehouse-outside databases corresponding to a warehouse-inside database according to a preset acquisition component, wherein the warehouse-inside database is a database inside a data warehouse, the warehouse-outside database is a database of a business system outside the data warehouse, and the warehouse-outside data synchronization information is used to represent data synchronization information between different warehouse-outside databases; analyzing the warehouse-outside data synchronization information to obtain warehouse-outside blood relationship metadata; generating a data blood relationship knowledge graph according to the warehouse-outside blood relationship metadata and the warehouse-inside database, wherein the data blood relationship knowledge graph is used to represent data blood relationship between the warehouse-inside database and the plurality of warehouse-outside databases.

2. The method of claim 1, wherein, The method for obtaining warehouse-outside data synchronization information of a plurality of warehouse-outside databases corresponding to a warehouse-inside database according to a preset acquisition component comprises: obtaining system architecture information of a plurality of business systems, and determining data interaction nodes corresponding to the plurality of warehouse-outside databases according to the system architecture information; installing the preset acquisition component on the data interaction nodes, and collecting data field mapping relationship, data format conversion protocol, data synchronization condition and data synchronization frequency corresponding to the data interaction nodes according to the preset acquisition component; determining the warehouse-outside data synchronization information according to the data field mapping relationship, the data format conversion protocol, the data synchronization condition and the data synchronization frequency.

3. The method of claim 1, wherein, The method for analyzing the warehouse-outside data synchronization information to obtain warehouse-outside blood relationship metadata comprises: analyzing the warehouse-outside data synchronization information by using a preset syntax rule to obtain a syntax analysis result; verifying the syntax analysis result according to a preset business rule to obtain a verification result; in the case that the verification result is verified, determining the warehouse-outside blood relationship metadata according to the syntax analysis result.

4. The method of claim 1, wherein, The method for generating a data blood relationship knowledge graph according to the warehouse-outside blood relationship metadata and the warehouse-inside database comprises: associating the warehouse-outside blood relationship metadata with warehouse-inside blood relationship metadata in the warehouse-inside database to obtain a metadata association result; determining business system data source entities, data warehouse table entities, synchronization channel relationships and synchronization dependency relationships according to a plurality of business systems; generating the data blood relationship knowledge graph according to the metadata association result, the business system data source entities, the data warehouse table entities, the synchronization channel relationships and the synchronization dependency relationships.

5. The method of claim 4, wherein, After generating the data blood relationship knowledge graph according to the warehouse-outside blood relationship metadata and the warehouse-inside database, the method further comprises: listening to interface change information of a plurality of business systems to obtain first change information; obtaining a data warehouse change log of the warehouse-inside database, and determining second change information according to the data warehouse change log; determining target change information according to the first change information and the second change information, and determining blood relationship change information according to the target change information and the metadata association result; updating the data blood relationship knowledge graph according to the blood relationship change information.

6. The method of claim 1, wherein, After generating the data blood relationship knowledge graph according to the warehouse-outside blood relationship metadata and the warehouse-inside database, the method further comprises: receiving demand change information for the business system; According to the requirement change information, deduction is performed in the data blood relationship knowledge graph, and a requirement deduction result is obtained; According to the requirement deduction result, a data warehouse change prediction result corresponding to the requirement change information is determined, wherein the data warehouse change prediction result is used to represent a database change operation to be performed by the in-warehouse database for the requirement change information.

7. A data lineage processing apparatus, comprising: Comprise: An information acquisition unit is configured to acquire, according to a preset acquisition component, out-warehouse data synchronization information of a plurality of out-warehouse databases corresponding to an in-warehouse database, wherein the in-warehouse database is a database inside a data warehouse, the out-warehouse database is a database of a business system outside the data warehouse, and the out-warehouse data synchronization information is used to represent data synchronization information between different out-warehouse databases; An information analysis unit is configured to analyze the out-warehouse data synchronization information to obtain out-warehouse blood relationship metadata; A graph generation unit is configured to generate a data blood relationship knowledge graph according to the out-warehouse blood relationship metadata and the in-warehouse database, wherein the data blood relationship knowledge graph is used to represent a data blood relationship between the in-warehouse database and the plurality of out-warehouse databases.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored executable program, wherein when the executable program is running, the computer readable storage medium controls the device where the computer readable storage medium is located to execute the data blood relationship processing method of any one of claims 1 to 6.

9. An electronic device, comprising: Comprise: A memory storing an executable program; A processor configured to run the program, wherein the program is executed to perform the data blood relationship processing method of any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the steps of the data blood relationship processing method of any one of claims 1 to 6.

Images