Data lineage analysis method, device, equipment and storage medium

Abstract

This application provides a data lineage resolution method, apparatus, device, and storage medium. The data lineage resolution method includes: acquiring script information to be parsed; replacing system variables with preset constants and user-defined functions with preset functions in the script information to obtain first information; pushing the associative structured query language in the first information onto a stack to obtain second information, wherein the associative structured query language includes intermediate structured query languages ​​and result structured query languages ​​that are mutually related during script runtime; constructing an abstract syntax tree based on the second information, wherein the abstract syntax tree includes the field relationships between nodes in the abstract syntax tree; and parsing the abstract syntax tree to obtain the data lineage relationship. This application can solve the problem of low accuracy in resolving data lineage relationships using traditional data lineage resolution methods.

Description

Technical Field

[0001] The present application relates to the field of big data analysis, and in particular to a data lineage analysis method, device, equipment and storage medium. Background Art

[0002] Lineage analysis is a technical means for comprehensive tracking of the data processing process, so as to find all the relevant metadata objects starting from a certain data object and the relationship between these metadata objects. The relationship between metadata objects specifically refers to the input and output relationship of the data flow representing these metadata objects. With the development and application of big data technology, the demand for data lineage analysis has emerged in the field of big data governance. By analyzing and processing the structured query language (SQL) that operates the database of the data middle platform, the mapping relationship between tables and fields between databases can be sorted out to form a data map, so as to track the flow of data in massive and huge data, check the source of upstream data and the destination of downstream data in the big data system, and facilitate big data developers and business analysts to grasp the consumption and source of data, so as to better manage the big data system and tap the value of data lineage.

[0003] The SQL types recorded in the data center operation log are diverse, and traditional data lineage analysis methods are difficult to parse all types of SQL, resulting in low accuracy in the parsed data lineage relationships. Summary of the invention

[0004] The embodiments of the present application provide a data lineage analysis method, device, equipment and storage medium, which can solve the problem of low accuracy of data lineage relationships analyzed by traditional data lineage analysis methods.

[0005] In a first aspect, an embodiment of the present application provides a data lineage analysis method, comprising:

[0006] Get the script information to be parsed;

[0007] The system variables in the script information to be parsed are replaced with preset constants, and the user-defined functions are replaced with preset functions to obtain first information;

[0008] Pushing the associated structured query language in the first information into a stack to obtain second information, wherein the associated structured query language includes an intermediate structured query language and a result structured query language that are associated with each other when the script is running;

[0009] constructing an abstract syntax tree according to the second information, wherein the abstract syntax tree includes a field association relationship between each node in the abstract syntax tree;

[0010] Parse the abstract syntax tree to obtain data lineage.

[0011] In one embodiment, the abstract syntax tree is parsed to obtain data lineage relationships, including:

[0012] According to the preset lineage analysis flow direction and the preset lineage analysis layer number, the field association relationship of each node in the abstract syntax tree is traversed and analyzed to obtain the data lineage relationship.

[0013] In one embodiment, the script information to be parsed includes a comment field, a process control field, and a variable assignment field; before replacing the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain the first information, the method further includes:

[0014] Remove the comment field, process control field and variable assignment field in the script information to be parsed.

[0015] In one embodiment, the method further comprises:

[0016] The data lineage relationship is sent to the data middle-office device through the real-time data service interface, and is used by the data middle-office device to display the data lineage relationship.

[0017] In one embodiment, the data lineage relationship includes at least one of the following:

[0018] Table relationships, field relationships, and qualification relationships.

[0019] In a second aspect, an embodiment of the present application provides a data lineage analysis device, comprising:

[0020] The acquisition module is used to obtain the script information to be parsed;

[0021] A replacement module, used to replace the system variables in the script information to be parsed with preset constants, and replace the user-defined functions with preset functions, to obtain the first information;

[0022] A stacking module, used for stacking the associated structured query language in the first information to obtain second information, wherein the associated structured query language includes an intermediate structured query language and a result structured query language that are associated with each other when the script is running;

[0023] A construction module, configured to construct an abstract syntax tree according to the second information, wherein the abstract syntax tree includes a field association relationship between each node in the abstract syntax tree;

[0024] The parsing module is used to parse the abstract syntax tree and obtain the data lineage relationship.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned data lineage analysis method when executed by the processor.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which is stored an implementation program for information transmission, and when the program is executed by a processor, the above-mentioned data lineage analysis method is implemented.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the above-mentioned data lineage analysis method.

[0028] The data lineage parsing method, apparatus, device and storage medium of the embodiments of the present application, after obtaining the script information to be parsed, replaces the system variables in the script information to be parsed with preset constants and replaces the custom functions with preset functions to obtain the first information, and then pushes the associated structured query language in the first information into a stack to obtain the second information. The script information to be parsed is converted into the second information through the above processing, so that the data association relationship in the script information to be parsed is converted into the second information that can be parsed by the abstract syntax tree construction algorithm, and then an abstract syntax tree is constructed according to the second information. The constructed abstract syntax tree is closer to the actual data association relationship when the script is running, and then the data lineage relationship is obtained by parsing the abstract syntax tree, so that the parsed data lineage relationship has a higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 It is a flowchart of a data lineage analysis method provided in an embodiment of the present application;

[0031] Figure 2 It is a structural schematic diagram of a data lineage analysis device provided in an embodiment of the present application;

[0032] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0034] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0035] The SQL types recorded in the data center operation log are numerous, and traditional data lineage parsing methods are difficult to parse all types of SQL, resulting in low accuracy of the parsed data lineage. For example, the method based on static job script parsing mainly covers static SQL scripts (including parameters) in the online code (Java, C language, etc.) version, SQL scripts (stored procedures, static SQL, views) and interpreted script programs (python, Perl, shell, etc.) running on the server. First, the SQL code is extracted. During the extraction process, the SQL code extraction module needs to remove dynamic parameters and non-SQL syntax codes to generate standard SQL business codes. Then, based on the standard SQL business code, the SQL syntax tree is constructed according to different database types to generate data lineage. This method cannot accurately identify the data processing relationship in the actual operation state, and the displayed data lineage is inconsistent with the production state lineage. For another example, the method based on job operation log analysis uses the online job production operation log as the source of blood relationship data analysis. Due to the large differences between different versions of the operation log, it is necessary to maintain different versions of log analysis strategies. In addition, the data middle platform operation logs come in various forms and the analysis success rate is low, resulting in a low accuracy rate in the blood relationship of the data obtained through analysis.

[0036] In order to solve the problems of the prior art, the embodiments of the present application provide a data lineage parsing method, device, equipment and storage medium. After obtaining the script information to be parsed, the embodiments of the present application replace the system variables in the script information to be parsed with preset constants and the custom functions with preset functions to obtain the first information, and then push the associated structured query language in the first information into a stack to obtain the second information. The script information to be parsed is converted into the second information through the above processing, so that the data association relationship in the script information to be parsed is converted into the second information that can be parsed by the abstract syntax tree construction algorithm, and then an abstract syntax tree is constructed according to the second information. The constructed abstract syntax tree is closer to the real data association relationship when the script is running, and then the data lineage relationship is obtained by parsing the abstract syntax tree, so that the parsed data lineage relationship has a higher accuracy. The data lineage parsing method provided by the embodiments of the present application is first introduced below.

[0037] Figure 1 FIG. 1 is a flow chart of a data lineage analysis method provided by an embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0038] S110, obtaining script information to be parsed.

[0039] The script information to be parsed can be obtained from heterogeneous databases, such as Oracle, MySQL, TeraData, TDH, GreenPlum, etc. The script information to be parsed can be obtained by reading the specified location of the parsing source configuration information in the heterogeneous database. The script information to be parsed can include table information related to the job script, operation log information, etc.

[0040] S120, replacing the system variables in the script information to be parsed with preset constants, and replacing the user-defined functions with preset functions, to obtain first information.

[0041] Among them, system variables and user-defined functions are usually not recognized by the abstract syntax tree construction algorithm during the abstract syntax tree construction process. System variables are variables that start with "${" and end with "}" in the script. Preset constants and preset functions can be pre-set. For example, the preset function can be selected as the UDF_FUNC() function. By replacing system variables with preset constants and user-defined functions with preset functions, the abstract syntax tree construction algorithm can recognize them, and the constructed abstract syntax tree is closer to the actual data association relationship when the script is running.

[0042] After obtaining the script information to be parsed, the embodiment of the present application replaces the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain first information, and then pushes the associated structured query language in the first information into a stack to obtain second information. The script information to be parsed is converted into the second information through the above processing, so that the data association relationship in the script information to be parsed is converted into the second information that can be parsed by the abstract syntax tree construction algorithm, and then an abstract syntax tree is constructed based on the second information. The constructed abstract syntax tree is closer to the actual data association relationship when the script is running, and then the data lineage relationship is obtained by parsing the abstract syntax tree, so that the parsed data lineage relationship has a higher accuracy.

[0043] S130, stacking the associated structured query language in the first information to obtain second information.

[0044] Among them, the first information includes associative structured query language and non-associative structured query language. The second information includes associative structured query language and non-associative structured query language after stacking. Associative structured query language includes intermediate structured query language and result structured query language that are mutually associated when the script is running. Non-associative structured query language does not change when the script is running. Associative structured query language refers to a type of information in which there are loop calls or SQL segment jump calls in the script. The number of loop calls or jump calls is determined by the runtime. The intermediate structured query language and result structured query language will be generated during the loop call or jump call process. Static scripts cannot obtain the information status at runtime. Stacking the associative structured query language can ignore the intermediate process of the associative structured query language in the subsequent abstract syntax tree construction, and only take the source and target pairing to improve the efficiency of abstract syntax tree construction.

[0045] S140: construct an abstract syntax tree according to the second information, where the abstract syntax tree includes a field association relationship between each node in the abstract syntax tree.

[0046] When constructing the abstract syntax tree, the abstract syntax tree can be constructed sentence by sentence according to the source script of the second information, the number of sentences to be parsed, and the start time. When constructing the abstract syntax tree, the abstract syntax tree can be constructed according to the second information by reading the execution engine Antlr4 grammar.

[0047] S150, parse the abstract syntax tree to obtain data lineage relationship.

[0048] In one embodiment, S150: parsing the abstract syntax tree to obtain data lineage relationships includes:

[0049] According to the preset lineage analysis flow direction and the preset lineage analysis layer number, the field association relationship of each node in the abstract syntax tree is traversed and analyzed to obtain the data lineage relationship.

[0050] Among them, the preset lineage analysis flow direction can be an upstream direction or a downstream direction, which can be set in advance. Each node is a node of the abstract syntax tree AST, and each node corresponds to a section of SQL syntax. For example, the AST includes target_table_columns (target table field node) and source_table_columns (source table field node). By traversing the next level nodes of the two nodes in sequence, the data lineage relationship between the source table field and the target field can be obtained. During the parsing process, for the associative structured query language after the stacking process, the stacking relationship can be pressed, and after linking the inner layer relationship, the direct mapping relationship between the source and the target can be extracted to generate the data lineage relationship. For the grammar that cannot be recognized temporarily, it is recorded in the queue to be learned as a whole group, pending subsequent learning and debugging, and configuring a new parsing template.

[0051] In one embodiment, the script information to be parsed includes a comment field, a process control field, and a variable assignment field. Before S120: replacing the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions, and obtaining the first information, the method further includes:

[0052] Remove the comment field, process control field and variable assignment field in the script information to be parsed.

[0053] The process control field may include statements such as GOTO, IF ERROR, and LOOP, and the comment field may include single-line comments: --, and multi-line comments: / ** / , etc. After elimination, the remaining script information to be parsed is all SQL.

[0054] In the embodiment of the present application, the system variables in the parsed script information are replaced with preset constants, and the user-defined functions are replaced with preset functions. Before obtaining the first information, the comment fields, process control fields, and variable assignment fields in the script information to be parsed are removed, and the remaining script information to be parsed is all SQL, which eliminates the noise information in the subsequent construction of the abstract syntax tree and improves the efficiency of constructing the abstract syntax tree.

[0055] In one embodiment, the data lineage relationship includes at least one of the following:

[0056] Table relationships, field relationships, and qualification relationships.

[0057] For example, the table relationship can be recorded as shown in Table 1:

[0058] Table 1

[0059] property Property Type Property Name Remark tableName String Table name alia String Table alias databaseName String Database Name isTempTable Boolean Is it a temporary table? tableLevel Integer Table Level

[0060] The field relationship can be recorded as shown in Table 2:

[0061] Table 2

[0062] property Property Type Property Name Remark tgtColumn Column Target Field srcColumn Column Source Field isUnrealSrc Boolean Is it a virtual source? srcOper String Operational Relationship

[0063] The limiting relationship can be recorded as shown in Table 3:

[0064] Table 3

[0065] property Property Type Property Name Remark tgtTable Table Target table srcColumn Column Limit Fields

[0066] By summarizing table relationships, field relationships, and restriction relationships, we can obtain a data result set, as shown in Table 4:

[0067] Table 4

[0068] property Property Type Property Name Remark relationTables List <relationtable> < / relationtable> Table relationship dataset relationColumns List <relationcolumn> < / relationcolumn> Field Relationship Dataset relationLimits List <relationlimit> < / relationlimit> Limiting Relationship Datasets

[0069] In one embodiment, the method further comprises:

[0070] The data lineage relationship is sent to the data middle-office device through the real-time data service interface, and is used by the data middle-office device to display the data lineage relationship.

[0071] The real-time data service interface includes an input interface and an output interface. The definitions of the input interface and the output interface can be shown in Table 5 and Table 6 respectively:

[0072] Table 5

[0073]

[0074]

[0075] The meanings of the parameters are as follows:

[0076] 1) srcSchemaName: the source database name, used to determine the ownership of the table and obtain the table ID.

[0077] 2) srcDBType: source database type, used to determine which database language to use for parsing.

[0078] 3)srcSQLScript: the content text of the ETL script.

[0079] 4) tableMap: stores the correspondence between all tables and their unique identifiers.

[0080] key: schemaName.table The full name of the table.

[0081] value: the unique identifier of the uuid table.

[0082] 5)columnMap: stores the correspondence between all tables and fields.

[0083] key: schemaName.table The full name of the table.

[0084] value: colList is a list of fields stored in order.

[0085] The information of each colInfo in columnList is as follows:

[0086] key: COLUMNNAME value: field name.

[0087] key: COLUMN_ID value: Field unique identifier UUID.

[0088] key: OBJ_ID value: The unique identifier UUID of the table corresponding to the field.

[0089] Table 6

[0090]

[0091] Among them, List <sqlresultvo>The analysis results are shown in Table 7:

[0092] Table 7

[0093] property Property Type Property Name Remark relaTables List <datarelatablevo> < / datarelatablevo> Table relationship dataset relaColumns List <datarelacolumnvo> < / datarelacolumnvo> Field Relationship Dataset relaLimits List <limitedrelacolumnvo> < / limitedrelacolumnvo> Limiting Relationship Datasets

[0094] When displaying data lineage relationships, the following information types can be displayed: table relationship view (DataRelaTableVO), as shown in Table 8; field relationship view (DataRelaColumnVO), as shown in Table 9; limited relationship view (LimitedRelaColumnVO), as shown in Table 10.

[0095] Table 8

[0096] property Property Type Property Name Remark SrcObjId String Source object ID TgtObjId String Target object ID DataRelaModify String Relationship modifiers DataRelaCode int Data Relationship Code AutoTypeCode int How to add a relationship DataRelaSrcId String Relationship Source Object DataRelaSrcName String Relationship source object name Const_Cd_Col_Id String Constant field ID Const_Value String Constant light value

[0097] Table 9

[0098] property Property Type Property Name Remark SrcObjId String Source Field ID TgtObjId String Target field ID DataRelaModify String Relationship modifiers DataRelaCode int Data Relationship Code AutoTypeCode int How to add a relationship DataRelaSrcId String Relationship Source Object RelaDesc String Field conversion relationship description DataRelaSrcName String Relationship source object name Const_Cd_Col_Id String Constant field ID Const_Value String Constant light value

[0099] Table 10

[0100]

[0101]

[0102] After obtaining the script information to be parsed, the embodiment of the present application replaces the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain first information, and then pushes the associated structured query language in the first information into a stack to obtain second information. The script information to be parsed is converted into the second information through the above processing, so that the data association relationship in the script information to be parsed is converted into the second information that can be parsed by the abstract syntax tree construction algorithm, and then an abstract syntax tree is constructed based on the second information. The constructed abstract syntax tree is closer to the actual data association relationship when the script is running, and then the data lineage relationship is obtained by parsing the abstract syntax tree, so that the parsed data lineage relationship has a higher accuracy.

[0103] Figure 1 The data lineage analysis method is described below. Figure 2 and attached Figure 3 Describe the device provided in the embodiment of the present application.

[0104] Figure 2 A schematic diagram showing the structure of a data lineage analysis device provided by an embodiment of the present application is shown. Figure 2 Each module in the device shown has the function of realizing Figure 1 The functions of each step in the process can achieve the corresponding technical effects. Figure 2 As shown, the device may include:

[0105] The acquisition module 210 is used to acquire the script information to be parsed;

[0106] A replacement module 220, used to replace the system variables in the script information to be parsed with preset constants, and replace the user-defined functions with preset functions, to obtain first information;

[0107] A stacking module 230 is used to stack the associated structured query language in the first information to obtain second information, wherein the associated structured query language includes an intermediate structured query language and a result structured query language that are associated with each other when the script is running;

[0108] A construction module 240 is used to construct an abstract syntax tree according to the second information, where the abstract syntax tree includes a field association relationship between each node in the abstract syntax tree;

[0109] The parsing module 250 is used to parse the abstract syntax tree to obtain data lineage relationships.

[0110] After obtaining the script information to be parsed, the embodiment of the present application replaces the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain first information, and then pushes the associated structured query language in the first information into a stack to obtain second information. The script information to be parsed is converted into the second information through the above processing, so that the data association relationship in the script information to be parsed is converted into the second information that can be parsed by the abstract syntax tree construction algorithm, and then an abstract syntax tree is constructed based on the second information. The constructed abstract syntax tree is closer to the actual data association relationship when the script is running, and then the data lineage relationship is obtained by parsing the abstract syntax tree, so that the parsed data lineage relationship has a higher accuracy.

[0111] In one embodiment, the parsing module 250 is specifically configured to:

[0112] According to the preset lineage analysis flow direction and the preset lineage analysis layer number, the field association relationship of each node in the abstract syntax tree is traversed and analyzed to obtain the data lineage relationship.

[0113] In one embodiment, the script information to be parsed includes a comment field, a process control field, and a variable assignment field; the apparatus further includes a removal module 260;

[0114] The elimination module 260 is used to eliminate the comment field, process control field and variable assignment field in the script information to be parsed before replacing the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain the first information.

[0115] In one embodiment, the apparatus further includes a sending module 270 for sending data lineage relationships to a data middle-office device via a real-time data service interface, so that the data middle-office device can display the data lineage relationships.

[0116] In one embodiment, the data lineage relationship includes at least one of the following:

[0117] Table relationships, field relationships, and qualification relationships.

[0118] After obtaining the script information to be parsed, the embodiment of the present application replaces the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain first information, and then pushes the associated structured query language in the first information into a stack to obtain second information. The script information to be parsed is converted into the second information through the above processing, so that the data association relationship in the script information to be parsed is converted into the second information that can be parsed by the abstract syntax tree construction algorithm, and then an abstract syntax tree is constructed based on the second information. The constructed abstract syntax tree is closer to the actual data association relationship when the script is running, and then the data lineage relationship is obtained by parsing the abstract syntax tree, so that the parsed data lineage relationship has a higher accuracy.

[0119] Figure 3 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the device may include a processor 301 and a memory 302 storing computer program instructions.

[0120] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0121] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 302 may include a removable or non-removable (or fixed) medium, or the memory 302 is a non-volatile solid-state memory. The memory 302 may be inside or outside the integrated gateway disaster recovery device.

[0122] In one example, the memory 302 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0123] The processor 301 reads and executes the computer program instructions stored in the memory 302 to implement Figure 1 The method in the embodiment shown in the figure is achieved Figure 1 The corresponding technical effects achieved by executing the method in the example shown are not repeated here for the sake of brevity.

[0124] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0125] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0126] Bus 310 includes hardware, software or both, and the components of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), Hyper Transport (Hyper Transport, HT) interconnection, industry standard architecture (Industry Standard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0127] The electronic device can execute the data lineage analysis method in the embodiment of the present application, thereby achieving Figure 1 The corresponding technical effects of the described data lineage analysis method.

[0128] In addition, in combination with the data lineage analysis method in the above embodiment, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data lineage analysis methods in the above embodiment is implemented.

[0129] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0130] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0131] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0132] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0133] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.< / sqlresultvo>

Claims

1. A data lineage analysis method, characterized in that: include: Get the script information to be parsed; The system variables in the script information to be parsed are replaced with preset constants, and the user-defined functions are replaced with preset functions to obtain first information; Pushing the associated structured query language in the first information into a stack to obtain second information, wherein the associated structured query language includes an intermediate structured query language and a result structured query language that are associated with each other when the script is running; constructing an abstract syntax tree according to the second information, wherein the abstract syntax tree includes a field association relationship between each node in the abstract syntax tree; The abstract syntax tree is parsed to obtain data lineage relationships.

2. The data lineage analysis method according to claim 1, characterized in that: The step of parsing the abstract syntax tree to obtain data lineage relationships includes: The field association relationship of each node in the abstract syntax tree is traversed and analyzed according to the preset lineage analysis flow direction and the preset lineage parsing layer number to obtain the data lineage relationship.

3. The data lineage analysis method according to claim 1, characterized in that: The script information to be parsed includes a comment field, a process control field, and a variable assignment field; before replacing the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain the first information, the method further includes: The comment field, the process control field and the variable assignment field in the script information to be parsed are removed.

4. The data lineage analysis method according to any one of claims 1 to 3, characterized in that: The method further comprises: The data lineage relationship is sent to the data middle-office device through a real-time data service interface, so that the data middle-office device displays the data lineage relationship.

5. The data lineage analysis method according to claim 1, characterized in that: The data lineage relationship includes at least one of the following: Table relationships, field relationships, and qualification relationships.

6. A data lineage analysis device, characterized in that: include: The acquisition module is used to obtain the script information to be parsed; A replacement module, used for replacing the system variables in the script information to be parsed with preset constants, and replacing the user-defined functions with preset functions, to obtain first information; A stacking module, used for stacking the associated structured query language in the first information to obtain second information, wherein the associated structured query language includes an intermediate structured query language and a result structured query language that are associated with each other when the script is running; A construction module, configured to construct an abstract syntax tree according to the second information, wherein the abstract syntax tree includes a field association relationship between each node in the abstract syntax tree; The parsing module is used to parse the abstract syntax tree to obtain data lineage relationship.

7. The data lineage analysis device according to claim 6, characterized in that: The analysis module is specifically used for: The field association relationship of each node in the abstract syntax tree is traversed and analyzed according to the preset lineage analysis flow direction and the preset lineage parsing layer number to obtain the data lineage relationship.

8. The data lineage analysis device according to claim 6, characterized in that: The script information to be parsed includes a comment field, a process control field and a variable assignment field; the device also includes a removal module; The elimination module is used to eliminate the comment field, process control field and variable assignment field in the script information to be parsed before replacing the system variables in the script information to be parsed with preset constants and the user-defined functions with preset functions to obtain the first information.

9. The data lineage analysis device according to any one of claims 6 to 8, characterized in that: The device also includes a sending module; The sending module is used to send the data lineage relationship to the data middle-office device through the real-time data service interface, and the data middle-office device is used to display the data lineage relationship.

10. The data lineage analysis device according to claim 6, characterized in that: The data lineage relationship includes at least one of the following: Table relationships, field relationships, and qualification relationships.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data lineage analysis method as claimed in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the data lineage analysis method according to any one of claims 1 to 5 is implemented.

13. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data lineage analysis method as described in any one of claims 1 to 5.

Images