Blood relationship index creating method, device and equipment, medium and product

Abstract

The invention provides a consanguinity index creation method, which can be applied to the field of big data, and comprises the following steps: obtaining a plurality of initial data, the initial data at least comprising job information and associated information representing a job flow path, and the plurality of initial data at least comprising initial data of different formats; according to the operation information and / or the associated information, multiple pieces of operation path information are constructed, and the operation path information represents the circulation relation between the operations; generating a first triple according to the job path information, wherein the first triple represents a circulation relationship among different fields in the initial data of the same format; according to the first triads, second triads with uniform structure expression are generated, and the second triads represent circulation relations among different fields in the initial data with the same format and / or different formats; and generating a blood relationship index according to each second triad. The invention further provides a device for executing the method, equipment, a medium and a program product.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a method, device, equipment, medium and product for creating a bloodline index. Background Art

[0002] As data assets become increasingly complex, field-level data lineage tracking has become a critical component of data governance. Existing technologies typically rely on parsing data processing scripts or collecting job and field dependency information through metadata management systems to establish lineage links. However, these approaches are often limited to systems operating on the same platform or with a unified structure. For heterogeneous systems involving multiple databases and configuration files, it is difficult to accurately restore the complete field flow relationship.

[0003] Especially in cross-platform data processing scenarios, field lineage relationships are often scattered across different types of job scripts and configuration files, making it difficult for existing methods to automatically integrate job links and field mapping information to form a unified field-level lineage structure. This not only affects the transparency of data flow but also limits the application of lineage information in scenarios such as operation and maintenance diagnosis and quality monitoring. Summary of the Invention

[0004] In view of this, the present application provides a method, device, electronic device, medium and program product for creating a bloodline index.

[0005] One aspect of the present application provides a method for creating a lineage index, including: obtaining multiple initial data, the initial data at least including job information and associated information representing the job flow path, the multiple initial data at least including initial data in different formats; constructing multiple job path information based on the job information and / or associated information, the job path information representing the flow relationship between jobs; generating a first triplet based on the job path information, the first triplet representing the flow relationship between different fields in the initial data of the same format; generating a second triplet with a unified structural expression based on each first triplet, the second triplet representing the flow relationship between different fields in the initial data of the same format and / or different formats; generating a lineage index based on each second triplet.

[0006] According to an embodiment of the present application, the associated information includes upstream and / or downstream dependencies of the job, and multiple job path information is constructed, including: constructing job path information based on the upstream and / or downstream dependencies of each job in different initial data.

[0007] According to an embodiment of the present application, the associated information includes the output information of the job and / or the input information of the job, and multiple job path information is constructed, including: constructing the job path information based on the respective output information and / or input information in different initial data.

[0008] According to an embodiment of the present application, the associated information includes the job name of the job and / or the path name of the job, and multiple job path information is constructed, including: constructing the job path information based on the job name of each job and / or the path name of each job in different initial data.

[0009] According to an embodiment of the present application, generating a first triplet based on the job path information includes: obtaining a field mapping relationship corresponding to the job path information based on the job name of the job; and generating the first triplet based on the field mapping relationship.

[0010] According to an embodiment of the present application, a first triplet is generated based on the job path information, including: matching the corresponding SQL file based on at least one of the job name, job number, script number, dependency, and table name of the job; obtaining the field mapping relationship corresponding to the job path information based on the SQL file; and generating the first triplet based on the field mapping relationship.

[0011] According to an embodiment of the present application, generating a second triplet with a unified structural expression includes: in two first triples, when the field names of the respective fields match, generating a second triplet with a unified structural expression according to the field names.

[0012] According to an embodiment of the present application, generating a second triplet with a unified structural expression includes: in two first triples, when the flow relationship of the field in one of them matches the field name of the field in the other first triplet, generating a second triplet with a unified structural expression based on the flow relationship and the field name.

[0013] According to an embodiment of the present application, the method also includes: there are overlapping fields between the two first triplets, and / or the two second triplets, and / or the first triplet and the second triplet, and the non-overlapping fields in the two first triplets, and / or the two second triplets, and / or the first triplet and the second triplet are constructed into a new second triplet.

[0014] According to an embodiment of the present application, a lineage index is generated based on each second triple, including: generating a field lineage knowledge graph based on each second triple; and generating a lineage index based on the field lineage knowledge graph.

[0015] According to an embodiment of the present application, the method also includes: obtaining the actual execution link and timing information of the job based on the job scheduling log of the job; and updating the field lineage knowledge graph based on the actual execution link and timing information.

[0016] Another aspect of the present application provides a bloodline index creation device, including: a first acquisition module, used to acquire multiple initial data, the initial data at least including job information and associated information representing the job flow path, and the multiple initial data at least including initial data of different formats; a first construction module, used to construct multiple job path information based on the job information and / or associated information, the job path information representing the flow relationship between jobs; a first generation module, used to generate a first triplet based on the job path information, the first triplet representing the flow relationship between different fields in the initial data of the same format; a second generation module, used to generate a second triplet with a unified structural expression based on each first triplet, the second triplet representing the flow relationship between different fields in the initial data of the same format and / or different formats; a third generation module, used to generate a bloodline index based on each second triplet.

[0017] Another aspect of the present application provides an electronic device comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the bloodline index creation method of any one of the aforementioned embodiments.

[0018] Another aspect of the present application provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method for creating a bloodline index according to any one of the aforementioned embodiments.

[0019] Another aspect of the present application provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the operation of the bloodline index creation method of any of the aforementioned embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0021] Figure 1 Schematically shows a flow chart of a method for creating a bloodline index according to an embodiment of the present application;

[0022] Figure 2 A block diagram schematically illustrates a device for creating a bloodline index according to an embodiment of the present application; and

[0023] Figure 3 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the technical solution of this application, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] An embodiment of the present application provides a method for creating a lineage index, including: obtaining multiple initial data, the initial data at least including job information and associated information representing the job flow path, the multiple initial data at least including initial data in different formats; constructing multiple job path information based on the job information and / or associated information, the job path information representing the flow relationship between jobs; generating a first triplet based on the job path information, the first triplet representing the flow relationship between different fields in the initial data of the same format; generating a second triplet with a unified structural expression based on each first triplet, the second triplet representing the flow relationship between different fields in the initial data of the same format and / or different formats; generating a lineage index based on each second triplet.

[0030] Figure 1 The flowchart of the method for creating a bloodline index according to an embodiment of the present application is schematically shown.

[0031] like Figure 1 As shown, the method for creating a bloodline index may at least include operations S110 to S150.

[0032] In operation S110 , a plurality of initial data are acquired, the initial data at least including job information and associated information representing a job flow path, and the plurality of initial data at least include initial data in different formats.

[0033] Initial data includes but is not limited to structured or semi-structured data content used to describe job scheduling, job dependencies, job parameters, field mapping relationships, and data flow paths. For example, initial data may include standard configuration files, XML files, SQL scripts, etc. The association information that represents the job flow path can be used to describe data items that connect jobs in terms of execution logic, data input and output, and dependencies. For example, association information may include but is not limited to upstream and downstream dependencies of jobs, path relationships between output files and input files, derivative mapping relationships of fields between jobs, and path identifiers in job execution parameters that represent data flow directions. Association information can be obtained by extracting and parsing meta-attributes such as job numbers, path parameters, and execution offset configurations.

[0034] Use structured parsing tools to process various configuration files, extract the number, name, scheduling frequency, execution offset, command parameters, data output location, data input source, field mapping and other contents or part of the contents of each job group, and normalize them into a unified data structure entry.

[0035] For example, the basic configuration of job group "0005" is extracted from the table file, including its scheduling period of "daily", dependence on upstream job "0003", output data file name "File 1", and data parameters used to control execution; the mapping definition between a file field and the target data table field is extracted from the markup language file to form a field derivation relationship; the field names involved in the field SELECT and INSERT statements are extracted from the script file for analyzing the field-level transfer chain.

[0036] It should be noted that in the various embodiments of the present application, file names, table names, data set names, contents of elements in triples, relevant parts of example SQL statements, etc. are all expressed in Chinese to illustrate their semantics or intrinsic meanings by way of example, but does not mean that Chinese is used in all specific implementations. In the subsequent content, this content will not be repeated. Those skilled in the art should be able to convert them into corresponding English characters or code statements according to common knowledge in the field during specific implementation to implement the solutions provided in the various embodiments of the present application.

[0037] In operation S120, multiple job path information is constructed based on the job information and / or associated information. The job path information represents the flow relationships between jobs. Specifically, the job path information represents the upstream and downstream logical relationships between multiple job units in the data processing flow. It can include job dependency topology, data output and consumption nodes, execution order, etc. By combining the dependency configuration, output paths, and input paths of each job group, a logical flow path between jobs is constructed.

[0038] For example, if job "0003" produces file "File 1", which is used as input by job "0005", the constructed path triple is <0003, derived to, 0005>; if job "0008" moves a file from directory "Directory 1" to directory "Directory 2", the constructed move path can be expressed as <0008, moved to, directory 2>.

[0039] In operation S130 , a first triplet is generated according to the operation path information, where the first triplet represents a flow relationship between different fields in the initial data of the same format.

[0040] The first triplet may be a structure of <source field, To, target field> that describes the derivation or mapping path between fields in the same format data source, and is used to characterize the basic unit of the lineage link at the field level.

[0041] Specifically, according to the field mapping rules extracted from the job path, the fields in the same source data format are mapped one by one, the derivative logic between the fields is extracted and structured as the first triplet, which serves as the most basic field relationship pair in the unified model.

[0042] For example, the field "cust_id" is extracted from the markup language file and derived from the intermediate file field to the target data table field "customer_id", and the triple is constructed as <cust_id, derived to, customer_id>; or, the field "region code" is extracted from the script file and passed from the source table to the intermediate dataset, and the first triple is constructed as <source table region code, derived to, intermediate dataset region code>.

[0043] In operation S140, a second triplet with a unified structural expression is generated based on each first triplet. The second triplet represents the flow relationship between different fields in the same format and / or different formats of the initial data. Specifically, the second triplet can be a unified expression result of the structure and semantics of the first triplet between different data formats. It is matched and integrated through rules such as field name, field meaning, and field order to ensure the connectivity and consistency of the derived paths between different source fields after conversion into the second triplet.

[0044] For example, if the field "customer_id" is derived from the source table to an intermediate file field, and then mapped from the field to the target data table field, multiple first triples can be merged into a unified triple <source table customer_id, derived to, target data table customer_id>.

[0045] In operation S150, a lineage index is generated based on each second triple. The lineage index may be a field lineage network index structure composed of the second triples, which is used to support field tracing, field destination analysis, abnormal path identification, semantic search, etc.

[0046] For example, all second triples are organized as nodes and edges in a graph data structure and stored in an index system with graph computing capabilities, thereby supporting query operations such as multi-hop analysis of field lineage paths, shortest path queries, and upstream and downstream field tracing.

[0047] According to the embodiment of the present application, through unified parsing and modeling of initial data in different formats, job path information and field triplet chains are constructed, and finally a lineage index with cross-platform, cross-format, and integrated structural characteristics is generated, realizing the structured expression of the entire process of field-level data flow, and effectively improving the lineage visualization, traceability, and controllability in data governance.

[0048] On the basis of the above-mentioned embodiment, the associated information includes upstream and / or downstream dependencies of the job, and the operation S120 may include operation S210.

[0049] In operation S210 , job path information is constructed based on upstream and / or downstream dependencies of respective jobs in different initial data.

[0050] Upstream dependencies can be one or more pre-jobs that a job needs to complete before it can execute. Downstream dependencies can be jobs whose output will be used as input or trigger conditions for one or more subsequent jobs after they are completed. Dependencies are generally declared explicitly through the dependency field in the job configuration file, or implicitly through scheduling configuration, execution order rules, etc.

[0051] Specifically, the dependency description fields in each job configuration are extracted from the initial data, and the logical relationship between jobs is modeled according to the dependency direction to form job path information in the form of line segments, trees, or directed graph structures. The job nodes are connected by dependency edges to indicate the job flow order.

[0052] For example, if the configuration file shows that job 0001 declares its downstream dependency as job 0003, the path <0001, derives to, 0003> can be established, indicating that 0003 can only be executed after 0001 completes, forming a sequential dependency relationship between the two. This path information does not currently involve field-level content; it only records the link relationship at the job level.

[0053] By systematically extracting and modeling the upstream and downstream dependencies of a job, a complete job execution path diagram can be quickly established, providing a clear process sequence basis for subsequent field lineage deduction, reducing the risk of path loss caused by complex job configuration or hidden dependencies, and improving the integrity and interpretability of the lineage model.

[0054] Based on the above embodiment, the associated information includes output information of the job and / or input information of the job, and operation S120 may include operation S310.

[0055] In operation S310 , operation path information is constructed based on the respective output information and / or input information in different initial data.

[0056] Output information can be the target objects generated during job execution, such as intermediate data files, structured tables, directory paths, or data units used by subsequent jobs. Input information can be the source data objects required by a job before it is started or executed, such as output results from upstream jobs, externally imported data, and configuration-defined input paths.

[0057] Specifically, by parsing the job parameter items recorded in various types of initial data, extracting the input sources and output targets represented therein, a data derivative path chain with "input-processing-output" as the basic unit is established between each job, and a job path information triple is generated.

[0058] For example, if the configuration for job "0007" defines its data source as dataset "Dataset 1" and its output file as "File 3.bin," the triple "0007, Output to, File 3.bin" can be constructed to indicate that job 0007 is responsible for exporting the data table to the specified file. If job "0008" moves the file from one directory to another, such as "Directory 3" → "Directory 4," the path triple "0008, Move to, Directory 4 / File 3.bin" can be constructed.

[0059] According to the embodiments of the present application, by extracting and structurally expressing the input and output information in the initial data, it is possible to model the actual data interaction link between jobs, thereby supporting the accurate attachment of field-level lineage to the path structure, effectively improving the consistency of lineage analysis results with the actual data flow, and avoiding logical fault problems caused by field names being out of the data flow path.

[0060] Based on the above embodiment, the associated information includes the job name of the job and / or the path name of the job, and operation S120 may include operation S410.

[0061] In operation S410 , job path information is constructed based on job names of respective jobs and / or path names of respective jobs in different initial data.

[0062] The job name can be a logical name used to identify a job unit as it is registered or displayed in the system. The path name is used to identify the data source path, data output path, execution script path, or scheduling path associated with the job. Specifically, the job configuration content in different initial data is parsed, and the name and path fields of each job are extracted. Through the semantic correspondence between job names or the structural inclusion relationship between path names, possible execution connection relationships between jobs are identified, and job path information representing the direction of data flow is constructed.

[0063] For example, if the path configuration of job 0008 records moving a file from the "directory 3" directory to the "directory 4" directory, the derived behavior in its job path can be identified and a path-level triple <0008, move to, directory 3 / file 3.bin> can be constructed, where "directory 3 / file 3.bin" is the target path name.

[0064] According to the embodiments of the present application, by parsing the job path information formed by the job name and the path name, it can assist in identifying the connection logic between jobs in the execution link when there is a lack of explicit dependency declarations or field mapping relationships, thereby improving the coverage and robustness of job flow modeling, and is particularly suitable for lineage repair and path completion in scenarios where the script chain is incomplete or the dependency chain is detached.

[0065] Based on the foregoing embodiments, operation S130 may include operations S510 to S520.

[0066] In operation S510, according to the job name of the job, obtain the field mapping relationship corresponding to the job path information. The field mapping relationship is used to describe the corresponding or derived relationship between the source field and the target field during the data processing, and the field mapping relationship can be established through information such as field names, field orders, and field types.

[0067] In operation S520, generate the first triple according to the field mapping relationship.

[0068] Specifically, by matching the path information associated with the job name, analyze the structure and corresponding relationship of each field in the data source and data target processed by the job, and then extract the field-level mapping relationship. Based on the extracted field mapping information, construct the derived triple between the source field and the target field and store it as the first triple.

[0069] For example, if a job processes a file named "file3.bin" and loads the field "cust_id" in it to the field "cust_id" of the target table "table1", then through the order matching relationship between the file structure and the target table field, construct the field mapping as: <cust_file3.bin:cust_id, derived to, table1.cust_id>, which is the first triple.

[0070] According to the embodiments of the present application, by establishing the mapping between the job path and the field structure with the job name as the index, it is possible to automatically extract the field lineage, avoid manually configuring the field relationships one by one, improve the efficiency and accuracy of lineage construction, and is particularly suitable for the field correspondence scenario between files and structured tables, and can support batch construction of field-level derived relationships.

[0071] Based on the foregoing embodiments, operation S130 may include operations S610 to S630.

[0072] In operation S610, match the corresponding SQL file according to at least one of the job name, job number, script number, dependency relationship, and table name of the job.

[0073] In operation S620, obtain the field mapping relationship corresponding to the job path information according to the SQL file.

[0074] In operation S630, generate the first triple according to the field mapping relationship.

[0075] SQL files represent structured scripts used to record field operation logic in data processing tasks. These files typically include data definition statements (such as table creation statements) and data operation statements (such as SELECT, INSERT, and JOIN). Static analysis can be used to extract field-level data source and target mappings. In this embodiment, the field mapping relationship can be the inter-field dependency path obtained by parsing the SQL statement.

[0076] For example, after obtaining job path information, a file matching operation is performed based on the job name, number, dependencies, or table names recorded in the path, retrieving the SQL script file associated with the job from the file repository. After file matching is complete, the SQL file is parsed to extract the field lists in statements such as SELECT and INSERT. Mappings between field inputs and outputs are established based on the structural position of the fields in the statements. Finally, based on the extracted field mappings, a field-level derivation relationship triple is constructed and recorded as the first triple.

[0077] For example, if the job path record for job number "0003" depends on the source table "Customer Basic Data Table" and the target path is the file "File 3.bin", the system matches the corresponding SQL file based on the number "0003" and parses the SQL statement as follows:

[0078] INSERT OVERWRITE DIRECTORY ' / path / file3.bin';

[0079] SELECT cust_id, region code FROM customer_base_data table;

[0080] The extracted field mapping relationships are: <customer base data table.cust_id, derived to, cust_file3.bin.cust_id> and <customer base data table.region code, derived to, cust_file3.bin.region code>, and are constructed into two first triples.

[0081] According to an embodiment of the present application, by matching the corresponding SQL file based on the job identification information and the data source table information, and parsing its field operation content, it is possible to automatically extract the field-level derivation relationship, solve the problem of missing or unclear description of field relationships in the configuration file, ensure the integrity and accuracy of the field lineage, and further enhance the lineage system's semantic perception of the processing logic.

[0082] Further, obtaining the field mapping relationship based on the SQL file may include: extracting the structured metadata information of the job group from the configuration file, and the metadata information includes, but is not limited to: job number, script path, job type, scheduling information, data source path, target path, and other contents. After obtaining the metadata information, the corresponding script path can be queried through the predefined configuration mapping rules, and the target SQL file can be loaded. The data definition statements and data operation statements included in the SQL file include, but are not limited to, table structure definition statements and data writing statements. After loading the SQL file, call the syntax analysis module to perform structured parsing on the script, identify the data reading source, data writing target, and field-level derived mapping relationship therein. The derived relationship may include field aliases, expression calculations, function calls, column position mappings, etc.

[0083] Based on the foregoing embodiments, operation S140 may include operation S710.

[0084] In operation S710, when the field names of the respective fields in the two first triples match, a second triple with a unified structural expression is generated according to the field name. The field name may be a logical name used to identify the field in its所属 data object. The second triple is a field-level derived relationship structure constructed under the unified semantic model, and its structural form is still <source field, derived to, target field>. However, this triple can span different jobs and initial data of different formats, and the unified expression of the field derivation path is achieved through structural normalization.

[0085] Specifically, after obtaining multiple first triples, the field names of the source fields and target fields in each triple are respectively extracted and normalized. If there are fields with the same field name in two triples, that is, the field names match, a unified field entity is established according to the field name, and the derived relationships originally belonging to different sources are merged into a second triple with a consistent structure.

[0086] For example, if there are a first triple <customer base data table.cust_id, derived to, cust_file3.bin.cust_id> and a first triple <cust_file3.bin.cust_id, derived to, table 1.cust_id>, the system identifies that the field name "cust_id" remains consistent in the intermediate field, and then constructs a second triple with a unified structure <customer base data table.cust_id, derived to, table 1.cust_id>, realizing the cross-job fusion of the field derivation chain.

[0087] According to the embodiments of the present application, by integrating multiple first triples based on field name matching, it is possible to achieve a normalized representation of the field lineage link across jobs and formats. In scenarios where the field names are consistent, the field transfer path can be accurately restored without relying on additional configuration information, thereby improving the automation degree and structural unity of lineage construction.

[0088] Based on the foregoing embodiments, operation S140 may include operation S810.

[0089] In operation S810, when the transfer relationships between the respective fields in two first triples match, a second triple with a unified structural expression is generated according to the transfer relationship. The matching of the transfer relationships between the respective fields may mean that the derivation direction, position order, or context structure between the fields in the two first triples have an equivalent or continuous relationship, so as to deduce the logical consistency on the field path.

[0090] Specifically, compare the field transfer paths in the two first triples. If there is equivalence in the transfer behavior between the two triples, such as the same field derivation order, one-to-one correspondence of field positions, or semantic structure with transfer continuity, it is considered that their transfer relationships match. Based on this matching relationship, the two original field links can be merged into a unified structural expression to form a second triple for representing the derivative path of the field.

[0091] For example, if there is a first triple <dataset1.customer_id, derived to, cust_file3.bin.customer_id>, and a first triple <cust_file3.bin.customer_id, derived to, table1.customer_id>, the system can match according to the transfer path direction, field order, and field semantics of the two triples, and determine that they have a continuous transfer relationship. Based on this matching relationship, a unified second triple <dataset1.customer_id, derived to, table1.customer_id> is constructed to complete the fusion of the field path.

[0092] According to the embodiments of the present application, by performing fusion processing between the first triples with matching transfer relationships, it is possible to achieve structural normalization of the field lineage relationship in scenarios where the field names are different but the data paths are continuous, enhance the sensitivity of the field tracking system to the transfer semantics of the fields, extend the lineage derivation ability to complex job chains with inconsistent names but continuous processes, and improve the coverage and practicality of lineage construction.

[0093] Based on the foregoing embodiments, operation S140 may include operation S910.

[0094] In operation S910, when the flow relationship of the fields in one of the two first triples matches the field name of the fields in the other first triple, a second triple with unified structural expression is generated according to the flow relationship and the field name. When the flow relationship of the fields in one of the two first triples matches the field name of the fields in the other first triple, it may be that in the case of no explicit naming of the source field, the source field is identified and bound only by relying on the position index of the field in the structure; a typical scenario is that the fields in an intermediate file (such as a binary file) lack explicit field names, and the corresponding relationship with the field names in the target table is established only through the structural definition order.

[0095] For example, after parsing the XML structure definition file, obtain the order of the bin file fields in the structure (such as the 1st field, the 2nd field, etc.), and bind it to the order of the field list in the target database table. For example, if the 3rd field in the bin file corresponds to "area code", an indirect name matching relationship can be established.

[0096] For example, if the first triple is <dataset1.area code, derived from, cust_file3.bin: data3>, and the other first triple is <cust_file3.bin: data3, derived from, table1.area code>, according to the structure definition in the XML configuration file, confirm that the field name corresponding to data3 in the bin file is "area code", then the unified second triple <dataset1.area code, derived from, table1.area code> can be constructed.

[0097] According to the embodiments of the present application, by combining the indirect matching relationship between the field order and the field name, it is possible to achieve field semantic alignment in an intermediate data file without explicit field identifiers, effectively solve the problem of field derivative recognition in the scenario of missing field names, and improve the expression integrity of the lineage path in the weakly structured intermediate link.

[0098] On the basis of the foregoing embodiments, the lineage index creation method may include operation S1010.

[0099] In operation S1010, when there are overlapping fields between two first triples, and / or two second triples, and / or a first triple and a second triple, the non-overlapping fields in the two first triples, and / or two second triples, and / or a first triple and a second triple are constructed into a new second triple.

[0100] Overlapping fields can refer to fields with the same name, equal field path, or structurally unified field reference relationship in two field triple structures. These fields are used to identify the intersection or connection node between two links. Non-overlapping fields can refer to the other end fields connected upstream and downstream of the intersection, that is, fields that may come from different source systems or different link paths.

[0101] When analyzing multiple field triplets (including the first triplet and the second triplet), it is identified that the target field is consistent with the source field of another triplet, that is, there are overlapping fields; at this time, the overlapping field is used as a connection bridge point, the non-overlapping field parts of the two triplets are extracted, and a new second triplet is constructed based on this, indicating that there is an indirect transmission relationship between the fields.

[0102] For example, if the first triplet A is <production group.product id, derived to, derived group_file3.bin.product id> and the second triplet B is <derived group_file3.bin.product id, derived to, product group.product id>, it is recognized that the field "derived group_file3.bin.product id" overlaps in both paths. Based on the derivation continuity of this field, the upstream and downstream paths can be automatically spliced ​​to construct a new second triplet <production group.product id, derived to, product group.product id> to express the complete link of this field from the source table to the target table.

[0103] According to the embodiments of the present application, by identifying and structurally splicing overlapping fields between field triplets, multiple field derivation paths can be effectively connected in series. When there are intermediate jumps, bridge fields or multiple operation chains in the path, cross-chain fusion and deep tracing of field lineage relationships can be achieved, thereby enhancing the integrity, scalability and automatic construction capabilities of the field lineage system under complex links.

[0104] Based on the foregoing embodiment, operation S150 may include operations S1110 to S1120.

[0105] In operation S1110, a field lineage knowledge graph is generated based on each second triple. The field lineage knowledge graph can be used to describe the semantic connection graph structure between fields based on the data processing path. Its nodes represent field entities, and its edges represent the derivation or inheritance relationship between fields. The structure is typically organized and stored using a graph database or graph structure model.

[0106] In operation S1120, a lineage index is generated based on the field lineage knowledge graph. The lineage index is used to refer to a queryable index structure generated based on the knowledge graph, which is used to quickly locate the source, destination, and transmission path of the field in the entire data chain.

[0107] First, the field entities and derivation relationships in multiple second triples are converted into nodes and directed edges in a graph structure. Each field is uniquely identified by its table and field name, and derivation relationships can be represented by semantic labels such as "derived from" and "generated from" to represent the connection between fields. Based on this graph structure, a field lineage index is constructed for fast retrieval. This index supports using any field as an entry node to retrieve its upstream and downstream field chains, transmission paths, and data processing operations involved in the field.

[0108] For example, if there are the following three second triplets:

[0109] 1) <dataset1.cust_id, derived to, firmware file_file3.cust_id>;

[0110] 2) <firmware file_file3.cust_id, derived to, table1.cust_id>;

[0111] 3) <Table 1.cust_id, generated from, 0005>;

[0112] Then you can construct the following nodes and edges in the graph:

[0113] Node: dataset1.cust_id, firmware_file3.cust_id, table1.cust_id;

[0114] Edge: Dataset1.cust_id → FirmwareFile_File3.cust_id (derived relationship),

[0115] Firmware file_file3.cust_id → table1.cust_id (derived relationship),

[0116] Table 1. cust_id → 0005 (generated relationship);

[0117] Based on this structure, a lineage index with "cust_id" as the keyword is generated, which supports field reverse query path, field impact analysis and upstream and downstream field chain tracking.

[0118] According to the embodiments of this application, by converting the second triple structure into a knowledge graph, a structured and standardized expression of field lineage relationships can be achieved, facilitating complex field tracking, path visualization, and semantic analysis operations. Furthermore, combining this with the lineage index constructed from the graph can significantly improve field query efficiency and visualization analysis capabilities.

[0119] Based on the above embodiment, the creation of the bloodline index may further include operations S1210 to S1220.

[0120] In operation S1210, the actual execution link and timing information of the job are obtained based on the job's job scheduling log. The job scheduling log represents the execution record generated during the job's execution, including fields such as the job ID, planned execution time, actual start and end times, execution platform, and running status. It is the source of operational data on field processing behavior in the time dimension. The actual execution link represents the execution sequence of each job node during the scheduling execution process, forming the job instance-level dependency topology. Timing information refers to the timestamp information generated during the actual operation of the job, which is used to describe the time boundaries or event flow of the field derivation path.

[0121] In operation S1220, the field lineage knowledge graph is updated based on the actual execution link and timing information. For example, the static job dependency graph G = (V, E) on which the field-level lineage graph depends is first obtained, where V represents the data processing unit (such as data conversion, synchronization, import, etc.) and E represents the dependency relationship between jobs. Subsequently, a set of target dates D is received. For each date T, the system retrieves the operation records of each job group based on the scheduling log, extracts metadata such as job number, platform type, start time, end time, and status, and constructs a time-state node v. J,T , and analyze its topological order to generate a dynamic dependency graph structure with time annotation and running status.

[0122] For example, first, based on the static dependency graph G = (V, E) output by the configuration parsing module, a topological sorting operation is performed on all job nodes in the graph to ensure that each node is scheduled for execution only when all its predecessor nodes have completed. Then, for the daily scheduling date T ∈ D, combined with the job instance records in the scheduling log, a temporal node v is constructed. i,T With the time state edge e i,j,T If job i is successfully executed on day T and job j is its direct dependent node, then a time edge is constructed:

[0123]

[0124] If j is not executed (MISSING), the delay attribute is set to empty, which is used for anomaly identification and lineage breakpoint marking.

[0125] In a data processing link, the scheduling topology path consists of three dependent job nodes: the data cleaning job on platform A, the file export job on platform B, and the data loading job on platform C. On April 1, 2024, in this link:

[0126] The first job started at 01:00 AM and completed at 01:15 AM;

[0127] The second job starts at 01:16 with a startup delay of 1 minute;

[0128] The third job was not executed on that day and is marked as "missing" by the system.

[0129] Based on the above actual execution information, the following timing edges in the processing link can be constructed:

[0130] Edge e1: indicates that after the first job is completed, the second job starts to execute after a delay of 1 minute;

[0131] Edge e2: represents the expected flow path after the second job is completed. However, due to the lack of downstream jobs, no effective execution duration and lineage connection are generated.

[0132] Use the logical node codes Job_A, Job_B, and Job_C instead of the specific job names (first job, second job, and third job) to construct the topological path:

[0133] e1 = Job_A (completed at 01:15) → [delayed 1 minute] → Job_B (started at 01:16);

[0134] e2 = Job_B (completed at 01:17:30) → [no actual connection] → Job_C (missing).

[0135] During the graph update phase, time-state nodes and their delay, status, and platform attributes are attached as operational labels to the job nodes and field links in the field lineage graph, forming an enhanced knowledge graph that integrates static structure and dynamic behavior. The links from source table processing to target table processing are annotated with temporal behavior attributes such as "processing time," "wait delay," and "failure jump point."

[0136] In addition, for multiple date ranges D = {T1, T2, …, T n The scheduling execution records in the} are aggregated and counted vertically to form a multi-day link indicator table. This includes indicators such as the average running time, average delay, task success rate, and number of failures of the job nodes on the field chain. For example:

[0137]

[0138] According to the embodiments of this application, by integrating scheduling topology and log data into a field lineage graph, the static derivation chain can be extended to an operational lineage structure, enabling field-level "temporal traceability," "behavioral interpretation," and "dynamic anomaly perception," thereby providing fundamental support for scheduling optimization, fault warning, and quality measurement. This graph not only preserves the derivative relationships between fields but also captures the actual transmission paths and state changes of fields at specific points in time.

[0139] According to an embodiment of the present application, a method for handling lineage anomalies is also provided. This method can be executed after the aforementioned configuration parsing and / or SQL parsing to perform online identification of structural errors, field inconsistencies, dependency errors, and other errors that arise during configuration parsing and SQL script parsing. This prevents non-compliant or unexplainable data from flowing into the subsequent field fusion and graph construction processes.

[0140] Identified anomalies are archived according to a unified structure, and the anomaly details are recorded in the form of JSON or database tables. Fields include anomaly type, source module, timestamp, job number, involved objects (such as table name or field name) and processing strategy, etc., to ensure subsequent traceability, statistics and playback.

[0141] Based on the repairability and severity of the exception, the system automatically determines whether to skip, delay processing, or interrupt the full-link build process for the abnormal object. For example, fields that can be deduced but not yet parsed are marked as "pending." Field-level builds are skipped if there are configuration errors or missing paths. If there are fatal errors such as loops in the dependency graph, the process is terminated and an alert is issued.

[0142] After the aforementioned configuration parsing and / or SQL script parsing operations, if the configuration file only records the source and target table names, the field structure is missing, and field-level mapping cannot be established, the job will be recorded as valid table-level mapping but "pending derivation" at the field level; enter the SQL module to perform delayed field binding; if the SQL parsing also fails, the "field structure missing" exception will be recorded, prompting manual re-entry.

[0143] After the aforementioned configuration parsing and / or SQL script parsing operations, the table name used in the job is not registered in the metadata repository, and its structure cannot be parsed. The exception type is recorded as "table name not registered"; the job number, time, and exception table name are recorded in the log; field-level parsing of this job is skipped.

[0144] After the aforementioned configuration parsing and / or SQL script parsing operations, if the file path definition used to describe the file is inconsistent across multiple configuration sources, for example, if a file path declared in a structure definition file has no corresponding generated path in the job scheduling configuration, resulting in a missing file source, the file will be marked as "unbound source" and recorded as a "path drift exception." Field-level binding for this path will be skipped, and an explicit prompt will be given in downstream loading or derivation scenarios, suggesting manual correction or configuration completion.

[0145] After the aforementioned configuration parsing and / or SQL script parsing, if the dependency relationships between jobs form a loop (for example, job A depends on job B, and job B depends on job A), this disrupts the directed acyclic graph structure and causes topological sorting failure. Topological analysis identifies dependency loops, records the exception type as "dependency cycle exception," aborts the current link building process, and generates an alert.

[0146] The embodiment of the present application also provides a method for constructing a field-level kinship knowledge graph, including: obtaining job path information and field mapping relationships, and uniformly converting them into standardized field processing records, wherein the field processing records include elements such as target fields, source field sets, field conversion types, and job identifiers, thereby achieving unified modeling expression of heterogeneous links. Combined with the exception information obtained by the exception handling method, all job instances involving "unsuccessful status" in the processing records are identified and filtered, and the corresponding field paths will be skipped or marked as "incomplete" and recorded in the diagnostic log; this step ensures that the field chain entering the graph construction stage has interpretability and structural integrity.

[0147] Furthermore, we can extract runtime meta-information for each job in the field processing chain, i.e., the actual link information, including but not limited to the job's start time, end time, and runtime platform attributes. This information is then associated with the field processing record, adding timing and runtime context attributes to the field path in the graph.

[0148] According to the preset domain semantic ontology specification, the field processing records are converted into a standardized triple set. The triples can include the following forms:

[0149] <target field, derived from, source field>: indicates that there is a derived relationship between fields;

[0150] <target field, generated from, job ID>: indicates that the target field is generated by a specific job;

[0151] <job ID, with start time, timestamp>, <job ID, on platform, platform type>: represents the runtime context attributes.

[0152] Each triple is then written into the graph database in batches, and an index structure is established for core predicates (such as derived from and generated from) to optimize the query performance based on the graph query language.

[0153] It provides visual interactive functions based on graph databases, supporting users to conduct multi-dimensional traceability queries according to field identifiers, job identifiers or time windows. The system automatically displays field paths in the form of a graph structure, and supports functions such as path highlighting, breakpoint diagnosis and job details linkage, thereby improving field lineage queries and data link observability.

[0154] Figure 2 The following schematically shows a block diagram of a device for creating a bloodline index according to an embodiment of the present application.

[0155] like Figure 2 As shown, the bloodline index creation device 1300 may include a first acquisition module 1310 , a first construction module 1320 , a first generation module 1330 , a second generation module 1350 , and a third generation module 1350 .

[0156] First acquisition module 1310 is used to acquire multiple initial data. The initial data includes at least job information and associated information representing the job flow path. The multiple initial data includes at least initial data in different formats. In some embodiments, first acquisition module 1310 can be used to perform operation S110 in the above-mentioned lineage index creation method, which is not further described here.

[0157] The first construction module 1320 is used to construct multiple operation path information based on the operation information and / or the associated information, where the operation path information represents the flow relationship between the operations. In some embodiments, the first construction module 1330 can be used to perform operation S120 in the above-mentioned lineage index creation method, which will not be described in detail here.

[0158] The first generation module 1330 is used to generate a first triplet based on the operation path information. The first triplet represents the flow relationship between different fields in the initial data of the same format. In some embodiments, the first generation module 1320 can be used to perform operation S130 in the above-mentioned lineage index creation method, which will not be described in detail here.

[0159] Second generation module 1340 is configured to generate second triples with a unified structural expression based on each first triple. The second triples represent the flow relationship between different fields in the initial data of the same format and / or different formats. In some embodiments, second generation module 1340 can be configured to perform operation S140 in the aforementioned method for creating a lineage index, and further description thereof is omitted here.

[0160] The third generation module 1350 is used to generate a lineage index according to each second triple. In some embodiments, the third generation module 1350 can be used to perform operation S150 in the above-mentioned lineage index creation method, which will not be described in detail here.

[0161] According to the embodiments of the present application, any number of modules, submodules, units, and subunits, or at least part of the functions of any number of them, can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, which can perform the corresponding functions when the computer program module is executed.

[0162] For example, any multiple of the first acquisition module 1310, the first construction module 1320, the first generation module 1330, the second generation module 1340, and the third generation module 1350 can be combined into one module / unit / sub-unit for implementation, or any one of these modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present application, at least one of the first acquisition module 1310, the first construction module 1320, the first generation module 1330, the second generation module 1340, and the third generation module 1350 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable means of integrating or packaging circuits, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the first acquisition module 1310, the first construction module 1320, the first generation module 1330, the second generation module 1340, and the third generation module 1350 may be at least partially implemented as a computer program module, which, when executed, may perform the corresponding function.

[0163] It should be noted that the data processing system part in the embodiments of the present application corresponds to the data processing method part in the embodiments of the present application. The description of the data processing system part specifically refers to the data processing method part and will not be repeated here.

[0164] Figure 3 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is schematically shown. Figure 3 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0165] like Figure 3 As shown, the electronic device 1400 according to an embodiment of the present application includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage portion 1408 into a random access memory (RAM) 1403. The processor 1401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1401 may also include onboard memory for caching purposes. The processor 1401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0166] Various programs and data required for the operation of the electronic device 1400 are stored in the RAM 1403. The processor 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. The processor 1401 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 1402 and / or the RAM 1403. It should be noted that the programs may also be stored in one or more memories other than the ROM 1402 and the RAM 1403. The processor 1401 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0167] According to an embodiment of the present application, electronic device 1400 may further include an input / output (I / O) interface 1405, which is also connected to bus 1404. Electronic device 1400 may also include one or more of the following components connected to I / O interface 1405: an input unit 1406 including a keyboard, mouse, etc.; an output unit 1407 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage unit 1408 including a hard disk; and a communication unit 1409 including a network interface card such as a LAN card or modem. Communication unit 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to I / O interface 1405 as needed. Removable media 1411, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1410 as needed, so that computer programs read from the removable media can be installed into storage unit 1408 as needed.

[0168] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1409, and / or installed from the removable medium 1411. When the computer program is executed by the processor 1401, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0169] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0170] According to embodiments of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0171] For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 1402 and / or the RAM 1403 described above and / or one or more memories other than the ROM 1402 and the RAM 1403 .

[0172] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the bloodline index creation method provided by the embodiment of the present application.

[0173] When the computer program is executed by the processor 1401, the above functions defined in the system / device of the embodiment of the present application are performed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0174] In one embodiment, the computer program can rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program can also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 1409, and / or installed from a removable medium 1411. The program code contained in the computer program can be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above. According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include but are not limited to Java, C++, Python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments of the present application may be combined and / or coupled in various ways, and all of these combinations and / or couplings fall within the scope of the present application.

[0176] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A method for creating a bloodline index, characterized in that: The method includes: Acquire a plurality of initial data, wherein the initial data at least includes job information and associated information representing a job flow path, and the plurality of initial data at least includes initial data in different formats; Constructing a plurality of operation path information according to the operation information and / or the association information, wherein the operation path information represents the flow relationship between the operations; generating a first triplet according to the operation path information, where the first triplet represents a flow relationship between different fields in the initial data of the same format; generating, based on each of the first triples, a second triplet with a unified structural expression, wherein the second triplet represents a flow relationship between different fields in the initial data of the same format and / or different formats; Generate a lineage index based on each of the second triples.

2. The method according to claim 1, characterized in that The associated information includes upstream and / or downstream dependencies of the job, and the construction of multiple job path information includes: The job path information is constructed based on upstream and / or downstream dependencies of each job in the different initial data.

3. The method according to claim 1, characterized in that The associated information includes output information of the job and / or input information of the job, and the constructing of multiple job path information includes: The operation path information is constructed based on the respective output information and / or the input information in the different initial data.

4. The method according to claim 1, wherein The associated information includes the job name of the job and / or the path name of the job, and the constructing of multiple job path information includes: The job path information is constructed based on the job name of each job and / or the path name of each job in the different initial data.

5. The method according to claim 1, wherein Generating a first triplet according to the operation path information includes: According to the job name of the job, obtaining the field mapping relationship corresponding to the job path information; The first triplet is generated according to the field mapping relationship.

6. The method according to claim 1, characterized in that Generating a first triplet according to the operation path information includes: Matching a corresponding SQL file according to at least one of the job name, job number, script number, dependency, and table name of the job; According to the SQL file, obtain the field mapping relationship corresponding to the job path information; The first triplet is generated according to the field mapping relationship.

7. The method according to any one of claims 1 to 6, characterized in that The generated structure expresses a unified second triple, including: In the case where the field names of the fields in the two first triples match, the second triples with unified structural expressions are generated based on the field names.

8. The method according to any one of claims 1 to 6, characterized in that The generated structure expresses a unified second triple, including: In the case where the flow relationships between the fields in the two first triples match, the second triplet with a unified structural expression is generated according to the flow relationships.

9. The method according to any one of claims 1 to 6, characterized in that The generated structure expresses a unified second triple, including: In the case where the flow relationship of the fields in one of the two first triples matches the field name of the field in the other first triple, a second triple with a unified structural expression is generated based on the flow relationship and the field name.

10. The method according to any one of claims 1 to 6, characterized in that The method further comprises: There are overlapping fields between two of the first triplets, and / or two of the second triplets, and / or between the first triplet and the second triplet. Two of the first triplets, and / or two of the second triplets, and / or non-overlapping fields in the first triplet and the second triplet are constructed into a new second triplet.

11. The method according to claim 1, characterized in that Generating a blood relationship index according to each of the second triples includes: Generate a field lineage knowledge graph based on each of the second triples; Generate a lineage index based on the field lineage knowledge graph.

12. The method according to claim 11, characterized in that The method further comprises: Obtaining actual execution link and timing information of the job according to the job scheduling log of the job; The field lineage knowledge graph is updated according to the actual execution link and the timing information.

13. A device for creating a bloodline index, characterized in that: The device includes: A first acquisition module is configured to acquire a plurality of initial data, wherein the initial data at least includes job information and associated information representing a job flow path, and the plurality of initial data at least includes initial data in different formats; A first constructing module is configured to construct a plurality of operation path information based on the operation information and / or the association information, wherein the operation path information represents a flow relationship between operations; A first generating module is configured to generate a first triplet according to the job path information, where the first triplet represents a flow relationship between different fields in the initial data of the same format; a second generating module, configured to generate, based on each of the first triples, a second triplet having a unified structural expression, wherein the second triplet represents a flow relationship between different fields in the initial data of the same format and / or different formats; The third generating module is used to generate a blood relationship index according to each of the second triples.

14. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, The method further comprises the step of executing the one or more computer programs to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images