A method and apparatus for generating a data lineage diagram

Abstract

This invention discloses a method and apparatus for generating a data lineage graph, relating to the field of data processing technology. One specific implementation of the method includes: determining the hierarchical structure of an initial lineage graph; segmenting the hierarchical initial lineage graph to obtain at least one initial lineage module; identifying the edge models of each of the at least one initial lineage module; using a density clustering algorithm to identify the key models of each of the at least one initial lineage module; and hiding other models in the initial lineage graph besides the key models and edge models to obtain a data lineage graph that highlights key models. This implementation, by constructing a data lineage graph that highlights key models, allows users to more intuitively understand the key models and processing links in the lineage, improving the understandability and practicality of the data lineage graph.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for generating a data lineage diagram. Background Technology

[0002] Data needs to be processed multiple times from its generation to its transformation into indicators or reports. Current big data platforms can automatically identify the data lineage based on the underlying Structured Query Language (SQL) processing logic and generate a data lineage diagram.

[0003] This data lineage diagram can display all tables involved in all intermediate processes in the processing chain, achieving a comprehensive and accurate display of the data. However, when an enterprise has a large number of data models and complex, multi-layered processing chains, the data lineage diagram, which displays the entire amount of data, suffers from poor readability when listing numerous tables. It fails to highlight key points, makes it difficult to quickly identify key models and locate them, and fails to demonstrate the distinction between hierarchical and categorized concepts of models, thus reducing the richness of information. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for generating a data lineage diagram. By highlighting key models in the data lineage diagram, users can more intuitively understand the key models and processing links in the lineage, thereby improving the comprehensibility and practicality of the data lineage diagram.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for generating a data lineage diagram is provided.

[0006] The generation of a data lineage diagram according to an embodiment of the present invention includes: Determine the hierarchical structure of the initial kinship graph, and segment the hierarchical initial kinship graph to obtain at least one initial kinship module; Identify the edge model of each of the at least one initial lineage module mentioned above; Density clustering algorithm is used to identify the key model of each initial lineage module in at least one of the above initial lineage modules; Hide the models other than the key model and the edge model in the initial kinship diagram to obtain the kinship diagram that highlights the key data.

[0007] Optionally, the method for obtaining the above-mentioned initial bloodline diagram includes: The initial stock model is labeled according to the preset model governance rules to obtain the stock model; The initial incremental model is named according to the preset model governance rules to obtain the incremental model; The existing and incremental models are analyzed to generate an initial kinship diagram.

[0008] Optionally, identifying the edge model of each of the at least one initial kinship modules includes: Based on the tags of the aforementioned existing models, the initial lineage module in which the aforementioned existing models reside is determined; The models in the initial lineage module containing the aforementioned existing models are sorted hierarchically to determine the edge models in the initial lineage module containing the aforementioned existing models.

[0009] Optionally, identifying the edge model of each of the at least one initial kinship modules includes: Identify the keywords in the table names of the above incremental models to determine the initial lineage module in which the above incremental models reside; The models in the initial lineage module containing the aforementioned incremental models are sorted hierarchically to determine the edge models in the initial lineage module containing the aforementioned incremental models.

[0010] Optionally, the above-mentioned key model for identifying each initial kinship module in at least one initial kinship module using density clustering algorithm includes: Density clustering algorithm was used to determine the model corresponding to the core points in the initial lineage module. The model corresponding to the core point of the aforementioned initial lineage module is taken as the key model of the aforementioned initial lineage module.

[0011] Optionally, the above-mentioned model for determining the core points in the initial lineage module using density clustering algorithm includes: For each model in the initial lineage module mentioned above: Use the above model as the center and draw a circle with a preset radius; Obtain the number of models in the circle mentioned above; If the number of models is greater than or equal to a preset threshold, the center of the circle is used as the core point to determine the model corresponding to the core point in the initial lineage module.

[0012] Optionally, when the number of models is greater than or equal to a preset threshold, the center of the circle is used as the core point to determine the model corresponding to the core point in the initial lineage module, including: In response to the above number of models being greater than or equal to a preset number threshold, the models in each circle are validated and validation results are generated. In response to the above verification result indicating that the verification failed, the above preset radius and the above preset quantity threshold are adjusted, and the circle is redrawn according to the adjusted preset radius and preset quantity threshold until the verification passes.

[0013] To achieve the above objectives, according to another aspect of the present invention, an apparatus for generating a data lineage diagram is provided.

[0014] An apparatus for generating a data lineage diagram according to an embodiment of the present invention includes: a segmentation module, used to determine the layers of an initial lineage diagram and segment the layered initial lineage diagram to obtain at least one initial lineage module; The first identification module is used to identify the edge model of each of the at least one initial lineage modules mentioned above; The second identification module is used to identify the key model of each initial lineage module in the at least one initial lineage module mentioned above using a density clustering algorithm; The hiding module is used to hide other models in the initial lineage diagram except for the key model and the edge model mentioned above, so as to obtain the lineage diagram with the key data displayed.

[0015] To achieve the above objectives, according to another aspect of the present invention, an electronic device for generating a data lineage diagram is provided.

[0016] An electronic device for generating a data lineage diagram according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a data lineage diagram generation method according to an embodiment of the present invention.

[0017] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0018] An embodiment of the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for generating a data lineage diagram according to an embodiment of the present invention.

[0019] One embodiment of the above invention has the following advantages or beneficial effects: by performing algorithmic identification on the models in the initial lineage diagram, key models are highlighted to obtain a key data lineage diagram. During the display process, users can more intuitively understand the key models and processing links in the lineage relationship, while hiding non-key models, focusing on the core of the key data lineage diagram, thus improving the understandability and practicality of the data lineage diagram. At the same time, the use of density clustering algorithm to identify key models realizes the automated identification of key models and reduces the cost of manual labeling.

[0020] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0021] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a flowchart illustrating a method for generating a data lineage diagram according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the method for obtaining an initial bloodline diagram according to an embodiment of the present invention; Figure 3 This is an example diagram of a data warehouse hierarchical structure according to an embodiment of the present invention; Figure 4 This is an example diagram of business classification according to an embodiment of the present invention; Figure 5 This is a schematic diagram of model segmentation and model localization according to an embodiment of the present invention; Figure 6 This is a flowchart illustrating the method for the model corresponding to the core points of an embodiment of the present invention. Figure 7 This is a schematic diagram illustrating the process of determining core points according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the main modules of the data lineage diagram generation device according to an embodiment of the present invention; Figure 9 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 10 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0022] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.

[0024] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0025] Figure 1 This is a schematic diagram of the main steps of the method for generating a data lineage diagram according to an embodiment of the present invention.

[0026] like Figure 1 As shown, the method for generating a data lineage diagram according to an embodiment of the present invention mainly includes the following steps: Step S101: Determine the layering of the initial bloodline diagram, and segment the layered initial bloodline diagram to obtain at least one initial bloodline module; Step S102: Identify the edge model of each initial lineage module in the at least one initial lineage module described above; Step S103: Use density clustering algorithm to identify the key model of each initial lineage module in at least one initial lineage module; Step S104: Hide the other models in the initial bloodline relationship diagram except for the key model and the edge model mentioned above, to obtain the bloodline relationship diagram with the key data displayed.

[0027] From its generation, processing, integration, circulation, to its eventual disappearance, new data inevitably maintains a connection with the original data before processing. This relationship between data is similar to kinship in human society and is referred to as data kinship. A data kinship diagram is a graph that illustrates data kinship, used to visually represent the source, flow, destination, and computational logic of data. Enterprises can use data kinship diagrams to gain a clearer understanding of the source and flow of data, thereby better managing and analyzing it. The initial kinship diagram in this invention is a kinship diagram generated using existing data kinship diagram generation methods. Any method capable of generating a data kinship diagram can be used, and no specific limitation is made here.

[0028] In one alternative embodiment, such as Figure 2 As shown, the method for obtaining the initial bloodline diagram includes the following steps: Step S201: Label the initial stock model according to the preset model governance rules to obtain the stock model; Step S202: Name the table of the initial incremental model according to the preset model governance rules to obtain the incremental model; Step S203: Analyze the existing model and the incremental model to generate an initial bloodline relationship diagram.

[0029] The existing and incremental models are determined based on the release time of the preset model governance rules. Big data models generated before the aforementioned preset model governance rules are called existing models; big data models generated after the aforementioned preset model governance rules are called incremental models. The preset model governance rules refer to the pre-defined rules for the standardized governance of models.

[0030] This paper applies digital warehouse theory to enterprise business operations, performing data warehouse layering and business classification. Based on this layering and classification, the layering and classification of existing models are determined, and the terminology of these models is standardized to obtain initial existing models. Then, the initial existing models are labeled according to pre-defined model governance rules. The accuracy and efficiency of this labeling depend on whether the pre-defined model governance rules align with the enterprise's actual development process and business scenarios. The higher the degree of matching between the pre-defined model governance rules and the enterprise's actual development process and business scenarios, the higher the accuracy and efficiency of the labeling.

[0031] As an example, data warehouse layering can be as follows: Figure 3 As shown, business categories can be as follows: Figure 4 As shown. As another example, for an initial stock model describing revenue details, it can be labeled as: stratification = FDM, subject = revenue.

[0032] The initial incremental model does not need to be labeled. Since the preset model governance rules have been published, the table names need to be named according to the preset model governance rules when the initial incremental model is generated to obtain the incremental model.

[0033] After obtaining the existing model and the incremental model, the data lineage diagram generation method in the existing technology can be used to generate the initial lineage diagram according to the SQL logic processing method of the base table in the existing technology.

[0034] In an optional embodiment, identifying the edge model of each of the at least one initial lineage modules includes: Based on the tags of the aforementioned existing models, the initial lineage module in which the aforementioned existing models reside is determined; The models in the initial lineage module containing the aforementioned existing models are sorted hierarchically to determine the edge models in the initial lineage module containing the aforementioned existing models.

[0035] In an optional embodiment, identifying the edge model of each of the at least one initial lineage modules includes: Identify the keywords in the table names of the above incremental models to determine the initial lineage module in which the above incremental models reside; The models in the initial lineage module containing the aforementioned incremental models are sorted hierarchically to determine the edge models in the initial lineage module containing the aforementioned incremental models.

[0036] The initial lineage diagram is layered and categorized according to two dimensions: data warehouse layering and business classification. The initial lineage diagram is then modularized in both horizontal and vertical directions to obtain at least one initial lineage module. Here, a binary array m is used as shown below. i,j This represents the set of all models in a certain initial lineage module, which is used to locate the initial lineage module and achieve module segmentation.

[0037]

[0038] Among them, x i This represents the hierarchical structure of the model collection within the data warehouse, such as the base data layer; y j This represents the category theme of the model set in business classification, such as revenue. The definition of this binary tuple allows for the modular segmentation of the initial lineage graph.

[0039] Based on the binary arrays described above, a dependency hierarchy is introduced, using a three-array w as shown below. i,j,r This represents a model within the initial lineage module, used for model localization.

[0040]

[0041] Where, x i This represents the hierarchical structure of the model collection within the data warehouse, such as the base data layer; y j This represents the category / topic of the model set within the business classification, such as revenue; z r This represents the dependency level of the model set it belongs to within the initial lineage graph, such as a level 2 dependency. This triple definition allows for unique model localization. As an example, module segmentation and model localization can be performed as follows: Figure 5 As shown.

[0042] Furthermore, dependency hierarchy refers to determining the hierarchy of a model based on the interdependencies between models. The dependency hierarchy of a model can be identified based on the initial lineage graph.

[0043] For each initial lineage module in at least one initial lineage module, sort all models in its model set according to dependency hierarchy, and then... z and max z As the edge model corresponding to the initial lineage module.

[0044] For existing models, their initial lineage module can be determined based on their labels; for incremental models, their initial lineage module can be determined by identifying keywords in the incremental model's table names. Therefore, incremental models do not require labeling or tag reading.

[0045] In an optional embodiment, the above-mentioned use of density clustering algorithm to identify the key model of each initial kinship module in the at least one initial kinship module includes: Density clustering algorithm was used to determine the model corresponding to the core points in the initial lineage module. The model corresponding to the core point of the aforementioned initial lineage module is taken as the key model of the aforementioned initial lineage module.

[0046] The model set in the initial lineage module is analyzed to identify key models. However, due to the lack of clear patterns in the processing complexity and processing links of different models, the initial number of key models in each initial lineage module cannot be determined. Clustering algorithms are algorithms that classify data points into specific groups based on their similarity. The k-means clustering algorithm is a partitioning-based clustering algorithm that requires determining initial cluster centers and is unreliable for circular and irregularly shaped sample points. Agglomerative hierarchical clustering algorithms have high computational complexity, resulting in high computational costs for massive enterprise data assets, and cannot handle outliers well. In contrast, density clustering algorithms can identify clusters and data objects within clusters by searching and analyzing local density peaks in the data space. It is an adaptive clustering algorithm that does not require pre-setting cluster centers, is insensitive to noise, and can discover clusters of arbitrary shapes. Therefore, this invention selects density clustering algorithm for identifying key models in the initial lineage module. As an example, the present invention may use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm.

[0047] After using density clustering algorithm to cluster the models in each initial lineage module, the center of each cluster is taken as the core point, the model corresponding to the core point is taken as the key model of the initial lineage module, and the key model is marked.

[0048] In terms of time sequence, since the existing models are models that existed before the release of the preset model governance rules, when using density clustering algorithm to cluster models in the initial lineage module, the existing models can be used for training to continuously optimize the algorithm parameters (preset radius and preset quantity threshold), thereby enabling the identification of whether the newly generated incremental models are key models and automatically generating data lineage relationships to improve the accuracy of key model identification and labeling.

[0049] In an optional embodiment, the above-mentioned use of density clustering algorithm to determine the model corresponding to the core points in the initial lineage module includes: For each model in the initial lineage module mentioned above: Use the above model as the center and draw a circle with a preset radius; Obtain the number of models in the circle mentioned above; If the number of models is greater than or equal to a preset threshold, the center of the circle is used as the core point to determine the model corresponding to the core point in the initial lineage module.

[0050] Specifically, such as Figure 6 As shown, for the model set of the initial lineage module, a model that has not been selected before is randomly selected as the center of a circle. A circle is drawn with a preset radius, and the number of models in the circle is obtained. The above steps are repeated until all models in the model set are traversed. It is then determined whether the number of models in each circle is greater than or equal to a preset threshold. In response to the number of models in the circle being greater than or equal to the preset threshold, the center of the circle is taken as the core point, and the model corresponding to the core point in the initial lineage module is determined.

[0051] Among them, the center of a circle that has at least one core point but does not satisfy the condition that the number of models in the circle is greater than or equal to a preset threshold is taken as a boundary point; while the center of a circle that does not have a core point and does not satisfy the condition that the number of models in the circle is greater than or equal to a preset threshold is taken as an outlier point.

[0052] As an example, such as Figure 7 As shown, when the preset radius is 1 and the preset number threshold is 3, a model is randomly selected from the model set as the initial cluster center A0. Using A0 as the center, a circle C is drawn with a radius of 1. The number of models in circle C is 4. Since 4 > 3, A0 is the core point. Following the above steps, all models in the model set are traversed, circles are drawn in turn, and it is determined whether the number of models in the circle is greater than or equal to 3, thereby determining the outliers, boundary points, and core points.

[0053] In an optional embodiment, when the number of models is greater than or equal to a preset threshold, determining the model corresponding to the core point in the initial lineage module by taking the center of the circle as the core point includes: In response to the above number of models being greater than or equal to a preset number threshold, the models in each circle are validated and validation results are generated. In response to the above verification result indicating that the verification failed, the above preset radius and the above preset quantity threshold are adjusted, and the circle is redrawn according to the adjusted preset radius and preset quantity threshold until the verification passes.

[0054] Furthermore, the models within each circle are validated by determining whether they align with the enterprise's data warehouse hierarchy and business classification, and by assessing their similarity to determine if they can be clustered into one category. When a model within a circle does not align with the data warehouse hierarchy or business classification, or / or the similarity between the models within the circle is low, a validation result indicating that the validation has failed is generated. The aforementioned preset radius and preset quantity threshold are then adjusted, and the circle is redrawn based on the adjusted preset radius and preset quantity threshold until the validation passes.

[0055] In an optional embodiment, after obtaining the highlighted kinship diagram, the above method may further include: The genealogy diagram of key displayed data is validated to determine whether the highlighted and hidden models align with the enterprise's business and data development. If deviations are observed, the preset model governance rules can be further refined, or the preset radius and quantity thresholds can be adjusted to redraw the circle and determine the core model. Through continuous optimization of algorithm parameters and preset model governance rules, the ability to identify models is enhanced, and the accuracy of model identification is improved.

[0056] According to the technical solution of this invention, by performing algorithmic identification on the models in the initial lineage diagram, key models are highlighted, resulting in a more realistic and readable data lineage diagram that emphasizes key data. During the display process, users can more intuitively understand the key models and processing links within the lineage, while hiding non-key models. This focuses on the core of the data lineage diagram, improving its comprehensibility and practicality. It helps enterprise business analysts gain a deeper understanding of the model processing process, business origins, and processing logic, better comprehending the data range of enterprise operating indicators, improving analysis efficiency, and reducing R&D pressure. It also helps enterprise data R&D personnel more clearly understand the data processing link and important processing stages, providing detailed and reasonable references for model modification and optimization, facilitating the shift of enterprise focus to key models. Simultaneously, the use of density clustering algorithms to identify key models achieves automated identification of key models, reducing manual labeling costs and enabling batch processing of models.

[0057] Furthermore, by employing pre-defined model governance rules to govern the models, and linking model governance with data lineage, a synergistic effect is achieved. Higher accuracy in model governance leads to higher precision in identifying key models, and verifying the identification results of key models helps improve model governance efficiency. For incremental models, tables can be created simply by following the pre-defined model governance rules, avoiding repeated model governance.

[0058] Figure 8 This is a schematic diagram of the main modules of the data lineage diagram generation device according to an embodiment of the present invention.

[0059] like Figure 8 As shown, the data lineage graph generation device 800 of this embodiment includes: a segmentation module 801, used to determine the layers of the initial lineage graph and segment the layered initial lineage graph to obtain at least one initial lineage module; a first identification module 802, used to identify the edge model of each initial lineage module in the at least one initial lineage module; a second identification module 803, used to identify the key model of each initial lineage module in the at least one initial lineage module using a density clustering algorithm; and a hiding module 804, used to hide other models in the initial lineage graph except for the key model and the edge model, to obtain a data lineage graph with key features.

[0060] In an optional embodiment of the present invention, the method for obtaining the initial lineage diagram includes: labeling the initial stock model according to preset model governance rules to obtain the stock model; naming the initial incremental model according to preset model governance rules to obtain the incremental model; and analyzing the stock model and the incremental model to generate the initial lineage diagram.

[0061] In an optional embodiment of the present invention, the first identification module 802 is further configured to: determine the initial lineage module in which the existing model is located based on the label of the existing model; sort the models in the initial lineage module in which the existing model is located according to the hierarchy, and determine the edge models in the initial lineage module in which the existing model is located.

[0062] In an optional embodiment of the present invention, the first identification module 802 is further configured to: identify keywords in the table name of the incremental model, determine the initial lineage module in which the incremental model is located; sort the models in the initial lineage module in which the incremental model is located according to the hierarchy, and determine the edge models in the initial lineage module in which the incremental model is located.

[0063] In an optional embodiment of the present invention, the second identification module 803 is further configured to use a density clustering algorithm to determine the model corresponding to the core point in the initial lineage module; and to use the model corresponding to the core point of the initial lineage module as the key model of the initial lineage module.

[0064] In an optional embodiment of the present invention, the above-mentioned use of density clustering algorithm to determine the model corresponding to the core point in the initial lineage module includes: for each model in the initial lineage module: drawing a circle with the model as the center and a preset radius; obtaining the number of models in the circle; and if the number of models is greater than or equal to a preset number threshold, taking the center of the circle as the core point to determine the model corresponding to the core point in the initial lineage module.

[0065] In an optional embodiment of the present invention, the above-mentioned use of density clustering algorithm to determine the model corresponding to the core point in the initial lineage module includes: for each model in the initial lineage module: drawing a circle with the model as the center and a preset radius; obtaining the number of models in the circle; and if the number of models is greater than or equal to a preset number threshold, taking the center of the circle as the core point to determine the model corresponding to the core point in the initial lineage module.

[0066] According to the technical solution of the present invention, by performing algorithmic identification on the models in the initial lineage diagram, key models are highlighted to obtain a key data lineage diagram. During the display process, users can more intuitively understand the key models and processing links in the lineage relationship, while hiding non-key models, focusing on the core of the key data lineage diagram, thus improving the comprehensibility and practicality of the data lineage diagram. At the same time, the density clustering algorithm is used to identify key models, realizing the automated identification of key models and reducing the cost of manual labeling.

[0067] Figure 9 An exemplary system architecture 900 is shown, to which the data lineage graph generation method or data lineage graph generation apparatus of embodiments of the present invention can be applied.

[0068] like Figure 9 As shown, system architecture 900 may include terminal devices 901, 902, and 903, network 904, and server 905. Network 904 is used as a medium to provide a communication link between terminal devices 901, 902, and 903 and server 905. Network 904 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0069] Users can use terminal devices 901, 902, and 903 to interact with server 905 via network 904 to receive or send data. Various communication client applications can be installed on terminal devices 901, 902, and 903, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platforms.

[0070] Terminal devices 901, 902, and 903 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0071] Server 905 can be a server that provides various services, such as a backend management server that supports data sent by users using terminal devices 901, 902, and 903. The backend management server can analyze and process the acquired data and feed back the processing results (such as highlighting a data lineage diagram) to the terminal devices.

[0072] It should be noted that the method for generating the data lineage diagram provided in this embodiment of the invention is generally executed by server 905, and correspondingly, the device for generating the data lineage diagram is generally located in server 905.

[0073] It should be understood that Figure 9 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0074] The following is for reference. Figure 10 It shows a schematic diagram of the structure of a computer system 1000 suitable for implementing a terminal device of the present invention. Figure 10 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0075] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1002 or programs loaded from storage section 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the system 1000. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0076] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0077] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs the functions defined above in the system of this invention.

[0078] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a segmentation module, a first identification module, a second identification module, and a hiding module. The names of these modules do not necessarily limit the module itself; for example, the acquisition module can also be described as "a module for determining the hierarchical structure of an initial kinship diagram and segmenting the hierarchical initial kinship diagram to obtain at least one initial kinship module."

[0081] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to include: determining the hierarchy of an initial lineage graph; segmenting the hierarchical initial lineage graph to obtain at least one initial lineage module; identifying edge models of each of the at least one initial lineage modules; using a density clustering algorithm to identify key models of each of the at least one initial lineage modules; and hiding other models in the initial lineage graph besides the key models and edge models to obtain a highlighted data lineage graph.

[0082] According to the technical solution of the present invention, by performing algorithmic identification on the models in the initial lineage diagram, key models are highlighted to obtain a key data lineage diagram. During the display process, users can more intuitively understand the key models and processing links in the lineage relationship, while hiding non-key models, focusing on the core of the key data lineage diagram, thus improving the comprehensibility and practicality of the data lineage diagram. At the same time, the density clustering algorithm is used to identify key models, realizing the automated identification of key models and reducing the cost of manual labeling.

[0083] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating a data lineage diagram, characterized in that, include: Determine the hierarchical structure of the initial kinship graph, and segment the hierarchical initial kinship graph to obtain at least one initial kinship module; Identify the edge model of each initial lineage module in the at least one initial lineage module; Density clustering algorithm is used to identify the key model of each initial lineage module in the at least one initial lineage module; Hide all models in the initial lineage graph except for the key model and the edge model to obtain a lineage graph that highlights the key data.

2. The method for generating a data kinship diagram according to claim 1, characterized in that, The method for obtaining the initial bloodline diagram includes: The initial stock model is labeled according to the preset model governance rules to obtain the stock model; The initial incremental model is named according to the preset model governance rules to obtain the incremental model; The existing and incremental models are analyzed to generate an initial kinship diagram.

3. The method for generating a data kinship diagram according to claim 2, characterized in that, The step of identifying the edge model of each of the at least one initial kinship modules includes: Based on the tags of the existing model, determine the initial lineage module in which the existing model resides; The models in the initial lineage module where the existing model is located are sorted according to their hierarchy to determine the edge models in the initial lineage module where the existing model is located.

4. The method for generating a data kinship diagram according to claim 2, characterized in that, The step of identifying the edge model of each of the at least one initial kinship modules includes: Identify keywords in the table name of the incremental model to determine the initial lineage module in which the incremental model resides; The models in the initial lineage module where the incremental model is located are sorted according to their hierarchy to determine the edge models in the initial lineage module where the incremental model is located.

5. The method for generating a data kinship diagram according to claim 1, characterized in that, The key model for identifying each initial kinship module in the at least one initial kinship module using a density clustering algorithm includes: Density clustering algorithm is used to determine the model corresponding to the core points in the initial lineage module; The model corresponding to the core point of the initial lineage module is taken as the key model of the initial lineage module.

6. The method for generating a data lineage diagram according to claim 5, characterized in that, The step of using density clustering algorithm to determine the model corresponding to the core points in the initial lineage module includes: For each model in the initial lineage module: Draw a circle with the model as the center and a preset radius; Obtain the number of models in the circle; If the number of models is greater than or equal to a preset threshold, the center of the circle is used as the core point to determine the model corresponding to the core point in the initial lineage module.

7. The method for generating a data kinship diagram according to claim 6, characterized in that, When the number of models is greater than or equal to a preset threshold, the model corresponding to the core point in the initial lineage module is determined by taking the center of the circle as the core point, including: In response to the number of models being greater than or equal to a preset threshold, the models in each circle are verified, and a verification result is generated. In response to the verification result indicating that the verification failed, the preset radius and the preset quantity threshold are adjusted, and the circle is redrawn according to the adjusted preset radius and preset quantity threshold until the verification passes.

8. An apparatus for generating a data lineage diagram, characterized in that, include: The segmentation module is used to determine the layers of the initial kinship graph, and to segment the layered initial kinship graph to obtain at least one initial kinship module. A first identification module is used to identify the edge model of each of the at least one initial lineage modules; The second identification module is used to identify the key model of each initial lineage module in the at least one initial lineage module using a density clustering algorithm; A hiding module is used to hide other models in the initial lineage diagram except for the key model and the edge model, so as to obtain a lineage diagram that highlights the data.

9. An electronic device for generating a data lineage diagram, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images