A method, device and equipment for constructing bloodline visualization data

Abstract

The present invention belongs to the field of bloodline data processing technology. In order to solve the complex problems of the existing bloodline data analysis process, a bloodline visualization data construction method, device and equipment are provided. The bloodline visualization data construction method includes sequentially splitting the sub-arrays in the bloodline array into several regions of interest, matching the keywords of each region of interest with the database keyword dictionary, extracting the table name and field combination; sequentially extracting the character string composed of the keywords of the corresponding region of interest with the table name and matching it with the inter-table data flow dictionary to obtain the data flow of the data table; splitting the field combination in the two data tables with data flow and matching them one by one in the order after splitting to obtain the inter-field data flow; constructing original bloodline data that can simultaneously represent table-level bloodline and field-level bloodline; reconstructing the original bloodline data into standard bloodline data and visualizing it. It can improve the speed of bloodline analysis and the multi-dimensional display of bloodline data.

Description

Technical Field

[0001] The present invention belongs to the technical field of bloodline data processing, and in particular relates to a method, device and equipment for constructing bloodline visualization data. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Data lineage analysis is the technical foundation for data value discovery and is widely used in the intermediary stages of data value discovery, completing tasks such as data flow analysis and data association analysis. Existing technologies for analyzing data lineage relationships typically extract information based on data structures extracted from an abstract syntax tree (AST). However, AST parsing typically relies on specific SQL statements and lacks the ability to parse extensions and functions that include specific databases. Furthermore, parsing complex SQL statements can be time-consuming.

[0004] In order to solve the above-mentioned problems of lack of parsing capability for extensions and functions containing specific databases and long parsing time, the existing technology provides a method of automatically parsing table-level and field-level lineage relationships of data using SQLParser. When SQLParser is used to parse lineage relationships, it constructs a feature library that can quickly identify database types, selects the correct database parsing method, parses the type of the current SQL script statement, and calls the corresponding parsing strategy according to different engine types. However, the table-level lineage relationship data parsing and the field-level lineage relationship data parsing are parsed separately, and the relationship between the field and the table name is not considered. This makes the parsing process complicated, and the lineage data cannot be directly displayed in a multi-dimensional lineage relationship. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a method, device and equipment for constructing bloodline visualization data, which can improve the speed of bloodline analysis and the multi-dimensional display of bloodline data.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides a method for constructing bloodline visualization data.

[0008] In one or more embodiments, a method for constructing bloodline visualization data is provided, including:

[0009] Obtain at least one SQL statement in the business system and perform word segmentation on it to obtain the corresponding sub-array, and then form a lineage array;

[0010] Sequentially split the sub-arrays in the lineage array into several regions of interest, match the keywords of each region of interest with the pre-built database keyword dictionary, and extract the table name and its field combination;

[0011] Sequentially extract the character strings composed of keywords corresponding to the area of ​​interest with the table name and match them with the pre-built inter-table data flow dictionary to obtain the data flow of the data table;

[0012] Based on the data flow direction of the data table and the field combination corresponding to the table name, the field combination in the two data tables with data flow direction is split and matched one by one in the order after splitting to obtain the data flow direction between the fields;

[0013] Based on the table name, data flow direction of the data table, and data flow between fields, construct original lineage data that can simultaneously represent table-level lineage and field-level lineage;

[0014] Determine the lineage node information and node connection information based on the original lineage data, and then reconstruct the original lineage data into standard lineage data; the standard lineage data includes a node set, an edge set, and a public set;

[0015] Visualize data lineage relationships based on standard lineage data.

[0016] As an implementation method, the inter-table data flow dictionary is composed of The flow direction is a binary character representing whether the data flows forward or backward.

[0017] As an implementation manner, the keywords in the database keyword dictionary are divided into three categories: field identification keywords, table identification keywords and other keywords.

[0018] As an implementation method, before sequentially splitting the sub-arrays in the lineage array into a plurality of regions of interest, the method further includes:

[0019] Eliminate redundant information in the lineage array. The process of removing redundant information is as follows:

[0020] Traverse the subarrays of the lineage array to check whether they contain fixed keywords:

[0021] When there is no inter-field association in the content after the fixed keyword, remove all the content from the fixed keyword index to the next keyword index in the subarray.

[0022] As an embodiment, the region of interest is The key-value pair format is stored; represents the i-th region of interest.

[0023] As an implementation method, the format of the original lineage data is: table name.field<-table name.field.

[0024] As an implementation method, when SQL statements are extracted from the execution log of the scheduling task, before the original lineage data is reconstructed into standard lineage data, the following steps are further included:

[0025] The order in which SQL statements are executed is used as the temporal relationship of the lineage data, and the timestamps of SQL statement execution are added to the lineage node information.

[0026] As an implementation method, the data lineage visualization includes table-level lineage visualization, field lineage visualization, and scheduling lineage visualization.

[0027] A second aspect of the present invention provides a device for constructing bloodline visualization data.

[0028] In one or more embodiments, a device for constructing bloodline visualization data includes:

[0029] A lineage array forming module is used to obtain at least one SQL statement in the business system and perform word segmentation processing on it to obtain corresponding sub-arrays, thereby forming a lineage array;

[0030] The database keyword matching module is used to sequentially split the sub-arrays in the lineage array into several regions of interest, match the keywords of each region of interest with the pre-built database keyword dictionary, and extract the table name and its field combination;

[0031] The inter-table data flow direction matching module is used to sequentially extract character strings consisting of keywords corresponding to the area of ​​interest with the table name and match them with the pre-built inter-table data flow direction dictionary to obtain the data flow direction of the data table;

[0032] The module for determining the data flow direction between fields is used to split the field combination in two data tables with data flow direction based on the data flow direction of the data table and the field combination corresponding to the table name, and to match them one by one in the order after splitting to obtain the data flow direction between fields;

[0033] The original lineage data construction module is used to construct original lineage data that can represent both table-level lineage and field-level lineage based on the table name, data flow direction of the data table, and data flow between fields;

[0034] A standard lineage data reconstruction module is used to determine lineage node information and node connection information based on the original lineage data, and then reconstruct the original lineage data into standard lineage data; the standard lineage data includes a node set, an edge set, and a public set;

[0035] The blood relationship visualization module is used to visualize data blood relationships based on standard blood relationship data.

[0036] A third aspect of the present invention provides an electronic device.

[0037] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for constructing bloodline visualization data as described above are implemented.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) The present invention performs word segmentation processing on the SQL statement to form a lineage array, and then sequentially splits the sub-arrays in the lineage array into several regions of interest, and then extracts the table name and its field combination by matching the keywords of each region of interest with the database keyword dictionary, and then matches the character string composed of the keywords of the corresponding region of interest with the table name with the inter-table data flow direction dictionary to obtain the data flow direction of the data table; based on the data flow direction of the data table and the field combination corresponding to the table name, the inter-field data flow direction is obtained, and the original lineage data that can simultaneously represent the table-level lineage and the field-level lineage is constructed, and finally the original lineage data is reconstructed into standard lineage data and the data lineage relationship is visualized. In this way, there is no need to analyze the fields separately, which improves the speed of lineage analysis, and the original lineage data can simultaneously represent the table-level lineage and the field-level lineage, realizing the multi-dimensional display of lineage data.

[0040] (2) The present invention uses self-built parsing rules based on the database keyword dictionary, inter-table data flow dictionary and lineage array to perform lineage relationship parsing of SQL statements. On the one hand, it can avoid the black box problem and insufficient database type scalability problem that may be faced when using public parsers. On the other hand, it has better parsing effects for implicit dependencies, conditional logic and multiple statements (such as multiple SQL statements in the ETL process) of specific SQL statements. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 1 is a flow chart of a method for constructing bloodline visualization data according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of a device for constructing bloodline visualization data according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of an electronic device according to an embodiment of the present invention;

[0045] Figure 4This is a flowchart of original bloodline construction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] Figure 1 FIG. 1 is a flow chart of a method for constructing blood relationship visualization data in an embodiment of the present invention. Figure 1 The method for constructing blood relationship visualization data in the embodiment shown may include the following steps S101 to S107.

[0050] The specific implementation process of steps S101 to S107 is as follows:

[0051] Step S101: Obtain at least one SQL statement in the business system and perform word segmentation processing on it to obtain corresponding sub-arrays, and then form a lineage array.

[0052] It should be noted here that the blood relationship data in the embodiment of the present invention is extracted from SQL statements.

[0053] There are two sources of SQL statements:

[0054] A plain text SQL statement that is used to fill in the SQL statement input in the scheduling task;

[0055] The other is the SQL statement execution record generated during the execution of the data processing scheduling task. In terms of lineage acquisition of the scheduling task, the execution log in the scheduling task records all the SQL statements of the task.

[0056] After collecting SQL statements from the business system, we also include: standardizing the string of QL statements, including unifying the string case, replacing the end symbol with a tag , replace line breaks, add and compress spaces, etc. This will ensure the accuracy of subsequent lineage visualization construction.

[0057] In the process of word segmentation processing of SQL statements, this embodiment uses the word segmentation tool Spacy to segment the string of SQL statements, and after the word segmentation is completed, the following A set of split two-dimensional arrays, i.e., lineage arrays Each independent SQL statement corresponds to a sub-item of the two-dimensional array. For example, the SQL string is a table creation statement.

[0058] It should be noted here that, in other embodiments, other existing methods may also be used to perform word segmentation processing on SQL statements.

[0059] Step S102: Sequentially split the sub-arrays in the lineage array into several regions of interest, match the keywords of each region of interest with a pre-built database keyword dictionary, and extract the table name and its field combination.

[0060] The lineage array is matched against the keyword dictionary. All matches in the keyword dictionary are extracted from the lineage array, and the array is split based on the matching position index to form the region of interest. For example, by removing blank arrays from the region of interest results, the table name and field information in the SQL statement can be obtained.

[0061] In this embodiment, before the sub-arrays in the lineage array are sequentially split into a plurality of regions of interest, the following steps are further included:

[0062] The process of removing redundant information from the lineage array is as follows:

[0063] Traverse the subarrays of the lineage array to confirm whether they contain fixed keywords (such as order by):

[0064] If there is no inter-field association after a fixed keyword, remove all content from the subarray from the fixed keyword index to the next keyword index. For example, the following SQL statement fragments can be ignored: "order by name asc" and "where id=12".

[0065] This can avoid subsequent dictionary matching errors caused by redundant information in the lineage array and improve matching efficiency and accuracy.

[0066] In the specific implementation process of step S102, the region of interest ROI (region of interesting )by The key-value pair format is stored; represents the i-th region of interest.

[0067] Each field combination contains all the fields in the table. Marked as .

[0068] During implementation, we collected keywords such as DDL, DML, and SYSTEM from various database types to form a database keyword dictionary. Keywords within the database keyword dictionary are divided into three categories: field identifier keywords, table identifier keywords, and other keywords. This allows for quick lookup of field and table identifiers using the database keyword dictionary.

[0069] The database keyword dictionary is the matching basis for extracting bloodline data. The database keyword dictionary contains all keywords for databases such as MySQL, Oracle, SQL Server, PostgreSQL, DAMO, Hangao, Shentong, and Renmin University of China Jincang.

[0070] Step S103: sequentially extracting character strings consisting of keywords corresponding to the region of interest with table names and matching them with a pre-built dictionary of inter-table data flow directions to obtain the data flow directions of the data tables.

[0071] Based on the specific database keyword combination and its structure in the entire SQL statement, the inter-table data flow in the current SQL statement can be determined, and the keyword combination that can represent the data flow is collected to build a data flow dictionary. The inter-table data flow dictionary is composed of The flow direction is a binary string representing the forward or backward flow of data. For example, the keyword combination "insert into select from" matches the data flow direction of "insert into table1 select * from table2."

[0072] It's important to note that for database types like HiveSQL, FlinkSQL, and SparkSQL that have additional unique keywords, when expanding these unused databases, we simultaneously supplement the keyword dictionary and inter-table data flow dictionary. This keyword dictionary is supplemented by querying the database's official documentation or system tables. New keywords typically represent new data operation modes. To supplement the inter-table data flow dictionary, we first extract key-value pairs for the data operation modes associated with the new keywords. Furthermore, we supplement data flow information based on the SQL logs of the new database type in the ETL system, enabling rapid expansion and compatibility of the data dictionary related to lineage analysis.

[0073] The inter-table data flow dictionary can be used to quickly match the data flow between tables.

[0074] Step S104: Based on the data flow direction of the data table and the field combination corresponding to the table name, the field combination in the two data tables with data flow direction is split and matched one to one in the order after splitting to obtain the data flow direction between the fields.

[0075] In this embodiment, the lineage data obtained when parsing SQL statements satisfies the lineage between tables, lineage between fields, and lineage flow information at the same time, avoiding the performance loss caused by additional SQL statement queries, and better reflecting the real data flow formed by SQL statement queries in the ETL system.

[0076] For example, to split a field combination by a set delimiter (usually a comma).

[0077] Step S105: Based on the table name, the data flow direction of the data table, and the data flow direction between fields, construct original lineage data that can simultaneously represent table-level lineage and field-level lineage.

[0078] The format of the original lineage data is: table name.field <-table name.field.

[0079] An example of raw lineage data is as follows:

[0080] Table1.col1<- Table2.col1<- Table3.col1;

[0081] Table1.col3<- Table2.col3<- Table3.col3;

[0082] Table4.*<-Table5.*.

[0083] Step S106: Determine the lineage node information and node connection information based on the original lineage data, and then reconstruct the original lineage data into standard lineage data; the standard lineage data includes a node set, an edge set, and a public set.

[0084] For field-level lineage display: Initially acquired lineage data is statistically recorded using field-level lineage. To address the lack of scalability in table-level and scheduling task-level lineage, lineage nodes are defined to unify data representation. Each lineage node is considered a collection of table information, including the table ID, table name, field ID, and field name.

[0085] In the data lineage analysis scenario for scheduling tasks, when SQL statements are extracted from the execution log of the scheduling task, before the original lineage data is reconstructed into standard lineage data, the following steps are also required:

[0086] The order in which SQL statements are executed is used as the temporal relationship of the lineage data, and the timestamps of SQL statement execution are added to the lineage node information.

[0087] An example of the structure of a bloodline node is shown in Table 1:

[0088] Table 1 Structure of bloodline nodes;

[0089]

[0090] Abstract the "<-" connections in the original lineage data into node connection definitions, including the connection ID, connection type, connection source, and connection target. The connection ID uniquely identifies the connection data, and connection types include table connections and field connections. Add table connections to lineage nodes with timestamps in chronological order, setting the connection source and target to the source table of the previous SQL statement and the source table of the current SQL statement, respectively.

[0091] The data reconstruction part converts the original bloodline data into a standard bloodline data in JSON format The data consists of three main parts: a node set, an edge set, and a public set. The node set stores lineage node data, the edge set stores node connection data, and the public set adds display parameters such as layout, node width and height, and color required for visualization.

[0092] Step S107: Visualize the data lineage relationship based on the standard lineage data.

[0093] Lineage data visualization is responsible for visualizing the reconstructed lineage data, providing lineage data parsing and rendering, and visual rendering based on SVG drawing. The data lineage visualization includes table-level lineage visualization, field lineage visualization, and scheduling lineage visualization.

[0094] In the visual rendering of lineage data, each lineage node data is rendered into an SVG element called a table node , the table node is expanded when visualizing the fields and scheduling lineage. When the table node is expanded, the internal field nodes are rendered as independent svg elements And render on the bloodline node Internal node The size is automatically adjusted.

[0095] The web page area for visualizing lineage data is defined as the canvas, based on the canvas size and lineage data The node width and height data described in the public description are added to the node grid. The node grid is set with a fixed multiple of the node width and height, and the center node is set. The automatic layout provides original, centered, and adaptable layouts. Taking the centered layout as an example, the rendering of the lineage data follows the order of rendering the upper left node of the canvas grid, and then the center node is translated to achieve the automatic layout. After the node rendering and layout are completed, The edge nodes connecting the data are drawn as SVG lines to complete the connection between the table nodes and the field nodes.

[0096] The lineage visualization interface displays data flow between tables, data processing links between fields, and table-level lineage for multiple SQL statements during scheduling. The data visualization uses a centered grid layout, and the canvas supports dragging, resizing, and borderless expansion. It also provides...

[0097] Node interaction logic: Clicking a visualized node automatically highlights associated upstream and downstream nodes and connections. Table-level lineage displays only table node information, with field information displayed in a pop-up window when the mouse hovers over the node. When displaying field lineage, the node simultaneously renders the table name and the list of associated fields, automatically resizing the node and redrawing the layout. When displaying scheduled lineage, node drawing is based on table-level lineage drawing, adding a virtual timeline to the canvas to map timing information.

[0098] Connection interaction logic: A connection is defined as a smooth curve with an arrow pointing to it. When the mouse moves over the connection, the SQL snippet that generates the association between nodes is displayed.

[0099] Global shortcut function extension: Visual canvas interface, providing quick search for node location, file import and export in canvas interface, and support for exporting to PNG format images.

[0100] In some optional embodiments, during the visualization of data lineage relationships, a layout template may be added, and relative position information of node rendering may be provided in advance in the layout template to reduce redrawing and rearrangement during the node rendering process.

[0101] like Figure 2 As shown, the bloodline visualization data construction device provided by the embodiment of the present invention can be implemented in software. The bloodline visualization data construction device includes the following software modules: a bloodline array formation module 201, a database keyword matching module 202, an inter-table data flow matching module 203, an inter-field data flow determination module 204, an original bloodline data construction module 205, a standard bloodline data reconstruction module 206 and a bloodline relationship visualization module 207.

[0102] The following is an introduction to the functions of each software module in the bloodline visualization data construction device:

[0103] A lineage array forming module 201 is used to obtain at least one SQL statement in the business system and perform word segmentation processing on it to obtain corresponding sub-arrays, thereby forming a lineage array;

[0104] The database keyword matching module 202 is used to sequentially split the sub-arrays in the lineage array into a number of regions of interest, match the keywords of each region of interest with a pre-built database keyword dictionary, and extract the table name and its field combination;

[0105] The inter-table data flow direction matching module 203 is used to sequentially extract character strings consisting of keywords corresponding to the region of interest with the table name and match them with a pre-built inter-table data flow direction dictionary to obtain the data flow direction of the data table;

[0106] The inter-field data flow direction determination module 204 is used to split the field combinations in two data tables with data flow directions based on the data flow directions of the data tables and the field combinations corresponding to the table names, and to match them one by one in the order after splitting, so as to obtain the inter-field data flow direction;

[0107] The original lineage data construction module 205 is used to construct original lineage data that can simultaneously represent table-level lineage and field-level lineage based on the table name, data flow direction of the data table, and data flow direction between fields;

[0108] The standard lineage data reconstruction module 206 is used to determine lineage node information and node connection information based on the original lineage data, and then reconstruct the original lineage data into standard lineage data; the standard lineage data includes a node set, an edge set, and a public set;

[0109] The blood relationship visualization module 207 is used to visualize the blood relationship of data based on standard blood relationship data.

[0110] It should be noted here that the various modules in the embodiment of the present invention correspond one-to-one to the steps in the bloodline visualization data construction method in the above embodiment, and the specific implementation process is the same, which will not be described in detail here.

[0111] The structure of the electronic device according to the embodiment of the present invention is described in detail below. Figure 3 The schematic diagram of the structure of the electronic device provided in the embodiment of the present invention can be understood as follows: Figure 3 Only exemplary structures of the electronic device are shown, not all structures. Part or all of the shown structures may be implemented as needed.

[0112] The electronic device provided by the embodiment of the present invention includes: at least one processor 301, a memory 302, a user interface 303 and at least one network interface 304. The various components in the bloodline visualization data construction device are coupled together through a bus system 305. It can be understood that the bus system 305 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 305 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 3 Various buses are labeled as bus system 305 .

[0113] The user interface 303 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0114] It will be appreciated that the memory 302 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The memory 302 in the embodiment of the present invention can store data to support the operation of the terminal. Examples of such data include any computer program used to operate on the terminal, such as an operating system and application programs. The operating system includes various system programs, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and handle hardware-based tasks. The application program may include various application programs.

[0115] In some embodiments, the bloodline visualization data construction device 300 provided in the embodiments of the present invention can be implemented using a combination of software and hardware. As an example, the bloodline visualization data construction device 300 provided in the embodiments of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the bloodline visualization data construction method provided in the embodiments of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0116] As an example, the processor 301 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0117] As an example of hardware implementation of the bloodline visualization data construction device 300 provided in an embodiment of the present invention, the device provided in an embodiment of the present invention can be directly executed by a processor 301 in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components to implement the bloodline visualization data construction method provided in an embodiment of the present invention.

[0118] The memory 302 in the embodiment of the present invention is used to store various types of data to support the operation of the bloodline visualization data construction device, or to store data for executing Figure 1 Examples of such data include any executable instructions for operating on a device for constructing bloodline visualization data, such as executable instructions. A program implementing the method for constructing bloodline visualization data from bloodline visualization data according to an embodiment of the present invention may be included in the executable instructions.

[0119] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion and / or installed from a removable medium. When the computer program is executed by the central processing unit, the various functions defined in the apparatus of the present application are performed.

[0120] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams and the combination of processes and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for constructing bloodline visualization data, characterized in that: include: Obtain at least one SQL statement in the business system and perform word segmentation on it to obtain the corresponding sub-array, and then form a lineage array; Sequentially split the sub-arrays in the lineage array into several regions of interest, match the keywords of each region of interest with the pre-built database keyword dictionary, and extract the table name and its field combination; Sequentially extract the character strings composed of keywords corresponding to the area of ​​interest with the table name and match them with the pre-built inter-table data flow dictionary to obtain the data flow of the data table; Based on the data flow direction of the data table and the field combination corresponding to the table name, the field combination in the two data tables with data flow direction is split and matched one by one in the order after splitting to obtain the data flow direction between the fields; Based on the table name, data flow direction of the data table, and data flow between fields, construct original lineage data that can simultaneously represent table-level lineage and field-level lineage; Determine the lineage node information and node connection information based on the original lineage data, and then reconstruct the original lineage data into standard lineage data; the standard lineage data includes a node set, an edge set, and a public set; Visualize data lineage relationships based on standard lineage data.

2. A method for constructing blood relationship visualization data according to claim 1, characterized in that: The inter-table data flow dictionary is composed of The flow direction is a binary character representing whether the data flows forward or backward.

3. The method for constructing blood relationship visualization data according to claim 1, wherein: The keywords in the database keyword dictionary are divided into three categories: field identification keywords, table identification keywords and other keywords.

4. The method for constructing blood relationship visualization data according to claim 1, wherein: Before sequentially splitting the sub-arrays in the lineage array into several regions of interest, the following is also included: Eliminate redundant information in the lineage array. The process of removing redundant information is as follows: Traverse the subarrays of the lineage array to check whether they contain fixed keywords: When there is no inter-field association in the content after the fixed keyword, remove all the content from the fixed keyword index to the next keyword index in the subarray.

5. The method for constructing blood relationship visualization data according to claim 1, wherein: The area of ​​interest is The key-value pair format is stored; represents the i-th region of interest.

6. The method for constructing bloodline visualization data according to claim 1, wherein: The format of the original lineage data is: table name.field <- table name.field.

7. The method for constructing blood relationship visualization data according to claim 1, wherein: When extracting SQL statements from the execution log of a scheduled task, before reconstructing the original lineage data into standard lineage data, the following steps are also included: The order in which SQL statements are executed is used as the temporal relationship of the lineage data, and the timestamps of SQL statement execution are added to the lineage node information.

8. The method for constructing blood relationship visualization data according to claim 1, wherein: The data lineage visualization includes table-level lineage visualization, field lineage visualization, and scheduling lineage visualization.

9. A device for constructing blood relationship visualization data, characterized in that: include: A lineage array forming module is used to obtain at least one SQL statement in the business system and perform word segmentation processing on it to obtain corresponding sub-arrays, thereby forming a lineage array; The database keyword matching module is used to sequentially split the sub-arrays in the lineage array into several regions of interest, match the keywords of each region of interest with the pre-built database keyword dictionary, and extract the table name and its field combination; The inter-table data flow direction matching module is used to sequentially extract character strings consisting of keywords corresponding to the area of ​​interest with the table name and match them with the pre-built inter-table data flow direction dictionary to obtain the data flow direction of the data table; The module for determining the data flow direction between fields is used to split the field combination in two data tables with data flow direction based on the data flow direction of the data table and the field combination corresponding to the table name, and to match them one by one in the order after splitting to obtain the data flow direction between fields; The original lineage data construction module is used to construct original lineage data that can represent both table-level lineage and field-level lineage based on the table name, data flow direction of the data table, and data flow between fields; A standard lineage data reconstruction module is used to determine lineage node information and node connection information based on the original lineage data, and then reconstruct the original lineage data into standard lineage data; the standard lineage data includes a node set, an edge set, and a public set; The blood relationship visualization module is used to visualize data blood relationships based on standard blood relationship data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the method for constructing bloodline visualization data as described in any one of claims 1 to 7 are implemented.

Images