Doll model identification method and device

Abstract

The invention relates to a doll model identification method and device. The method comprises the following steps: acquiring service data of a service scene, and extracting data main body features from the service data; acquiring a data processing script of the service scene; according to the data main body features and the service data, the data processing script is analyzed to obtain a syntax tree, the syntax tree is traversed, a blood relationship knowledge graph is constructed, and the blood relationship knowledge graph is used for indicating a dependency relationship between graph nodes and processing logic; marking data subjects for the blood relationship knowledge graph to enable each graph node to carry subject tracing information of the corresponding data subject; according to the marked consanguinity knowledge graph and the syntax tree, a transformation significance evaluation value is determined, and the transformation significance evaluation value is used for reflecting the transformation degree of each time of data transformation; and according to the marked consanguinity knowledge graph and the transformation significance evaluation value, determining a doll model identification result. According to the scheme provided by the invention, automatic and accurate identification of the data main body can be realized.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method and apparatus for recognizing matryoshka doll patterns. Background Technology

[0002] With the rapid development of big data and artificial intelligence, data platforms are widely used in enterprise-level scenarios. Users typically generate thematic data tables or models that meet business needs through SQL queries, data development scripts, or visual modeling tools.

[0003] Currently, the "nested doll" model refers to a data model where users repeatedly concatenate and trim fields to generate new tables or models, resulting in the same entity (such as a user behavior table, order table, log table, etc.) being nested and referenced multiple times. Over time, this creates a data model with excessively long chains and layers of dependencies. However, while big data platforms typically provide task orchestration (DAG), metadata management, and lineage analysis functions, they cannot detect the repeated concatenation problem in nested doll models. Therefore, related technologies lack automated and accurate methods for identifying data entities, making it difficult to identify the business boundaries of nested doll models in multi-entity mixed scenarios. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a method and apparatus for recognizing matryoshka doll models, which can achieve automated and accurate recognition of data subjects.

[0005] The first aspect of this application provides a method for recognizing nesting doll patterns, including: Acquire business data from a business scenario and extract data subject features from the business data, wherein the business data includes business semantic description data of the business data table and data table features; Obtain the data processing scripts related to the business scenario; Based on the data subject characteristics and the business data, the data processing script is parsed to obtain a syntax tree, and the syntax tree is traversed to construct a lineage knowledge graph. The lineage knowledge graph is used to indicate the dependencies and processing logic between graph nodes, and the graph nodes include data entity nodes and / or process logic nodes. The bloodline knowledge graph is labeled with data subjects, so that each graph node carries the subject tracing information of the corresponding data subject; Based on the labeled kinship knowledge graph and the syntax tree, a transformation significance evaluation value is determined, which is used to reflect the degree of transformation of each data transformation; The recognition result of the matryoshka doll model is determined based on the labeled kinship knowledge graph and the transformation significance evaluation value.

[0006] In one possible implementation, the data subject characteristics include the data subject and the subject identifier and mapping relationship data corresponding to the data subject, wherein the mapping relationship data is used to indicate the association between the business data table and the data subject; The step of parsing the data processing script based on the data subject characteristics and the business data to obtain a syntax tree, and then traversing the syntax tree to construct a lineage knowledge graph, includes: The data processing script is converted into a syntax tree using a preset parser generator. The syntax tree is recursively traversed to extract the lineage relationship tuples; Based on the lineage relationship tuple, a column-level lineage relationship diagram is constructed. The column-level lineage relationship diagram is used to indicate the source information, destination information and the data body to which each field in the business data table belongs. Based on the column-level kinship diagram, a kinship knowledge graph is constructed.

[0007] In one possible implementation, the subject tracing information is used to indicate the subject identifier of at least one data subject corresponding to the upstream graph node of each graph node, and there is a calling relationship between the at least one data subject and the data subject corresponding to the graph node.

[0008] In one possible implementation, determining the transformation significance evaluation value based on the labeled kinship knowledge graph and the syntax tree includes: Based on the labeled lineage knowledge graph and the syntax tree, the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value are determined. The structural change degree evaluation value is used to reflect the degree of change of the field set, the logical complexity evaluation value is used to reflect the complexity of the data processing logic, and the association breadth evaluation value is used to reflect the association complexity of the data transformation. Assign a first weight to the structural change degree evaluation value, a second weight to the logical complexity evaluation value, and a third weight to the association breadth evaluation value; Based on the first weight, the second weight, and the third weight, the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value are weighted and fused to obtain the transformation significance evaluation value.

[0009] In one possible implementation, the structural change assessment value is determined through the following steps: Based on the list of source tables extracted from the FROM clause of the syntax tree, determine the total number of input fields involved in the data transformation; The final number of output fields in the SELECT list is determined based on the output field items extracted from the SELECT list in the syntax tree; The number of new fields is determined based on the column-level lineage diagram, the SELECT list of the syntax tree, and the number of output fields. The number of fields to be modified is determined based on the syntax tree, the number of newly added fields, the number of output fields, and the column-level lineage diagram. The number of fields to be passed through is determined based on the number of output fields, the number of newly added fields, and the number of modified fields. Based on the column-level lineage diagram, determine the input fields referenced by the output, which will be the number of input fields used. The number of fields to be deleted is determined based on the total number of input fields and the number of input fields used. The total number of fields that have changed is determined based on the number of newly added fields, the number of modified fields, and the number of deleted fields; Based on the total number of input fields and the number of newly added fields, determine the total field space involved in the data transformation; The structural change assessment value is determined based on the total number of fields that have changed and the total field space.

[0010] In one possible implementation, the logical complexity evaluation value is determined through the following steps: Traverse all function calls in the syntax tree, sum the called functions by weight, and obtain the function complexity score; Extract the corresponding logical structure based on the WHERE and HAVING clauses of the syntax tree; Based on the logical structure and the preset predicate recognition rules, determine the total number of predicates and the maximum nesting depth of logical operators; The predicate complexity score is determined based on the total number of predicates and the maximum nesting depth. The logical complexity evaluation value is determined based on the function complexity score and the predicate complexity score.

[0011] In one possible implementation, the association breadth evaluation value is determined through the following steps: Based on the column-level lineage diagram and the FROM and JOIN clauses of the syntax tree, determine the total number of tables involved in the data transformation; The association complexity score is determined based on the column-level lineage graph, the JOIN clauses of the syntax tree, the total number of tables, and the preset JOIN operation weight dictionary. The association breadth evaluation value is determined based on the association complexity score.

[0012] In one possible implementation, the values ​​of the first weight, the second weight, and the third weight are associated with the scenario profile of the business scenario.

[0013] In one possible implementation, determining the matryoshka doll model recognition result based on the labeled kinship knowledge graph and the transformation salience evaluation value includes: Traverse the marked kinship knowledge graph and mark data transformation operations with transformation significance evaluation values ​​lower than a preset significance threshold as high inertia transformation operations; In the labeled lineage knowledge graph, a long link consisting of multiple consecutive high-inertia transformation operations is searched, and the long link is identified as a nested model.

[0014] A second aspect of this application provides a matryoshka doll model recognition device, comprising: The business data acquisition module is used to acquire business data of a business scenario and extract data subject features from the business data. The business data includes business semantic description data of the business data table and data table features. The script acquisition module is used to acquire data processing scripts related to the business scenario. The knowledge graph construction module is used to parse the data processing script based on the data subject characteristics and the business data to obtain a syntax tree, and traverse the syntax tree to construct a lineage knowledge graph. The lineage knowledge graph is used to indicate the dependencies and processing logic between graph nodes. The graph nodes include data entity nodes and / or process logic nodes. The subject tagging module is used to tag the data subjects in the bloodline knowledge graph, so that each graph node carries the subject tracing information of the corresponding data subject; The transformation significance evaluation module is used to determine the transformation significance evaluation value based on the labeled kinship knowledge graph and the syntax tree. The transformation significance evaluation value is used to reflect the degree of transformation of each data transformation. The identification module is used to determine the identification result of the matryoshka model based on the labeled kinship knowledge graph and the transformation significance evaluation value.

[0015] The technical solution provided in this application may include the following beneficial results: By acquiring business data from a business scenario and extracting data subject features from the business data (including business semantic description data and data table features), data processing scripts related to the business scenario are obtained. Then, based on the data subject features and business data, the data processing scripts are parsed to obtain a syntax tree. The syntax tree is traversed to construct a lineage knowledge graph. The lineage knowledge graph is used to indicate the dependencies and processing logic between graph nodes. The graph nodes include data entity nodes and / or process logic nodes. The lineage knowledge graph is labeled with data subjects, so that each graph node carries the subject tracing information of the corresponding data subject. Then, based on the labeled lineage knowledge graph and the syntax tree, a transformation significance evaluation value is determined. This transformation significance evaluation value reflects the degree of change in each data transformation. Based on the labeled lineage knowledge graph and the transformation significance evaluation value, the nested model recognition result is determined. This enables automated and accurate identification of data subjects, realizes the identification of nested model business boundaries in multi-subject mixed scenarios, reduces waste of computing and storage resources, and improves data analysis and recognition efficiency.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0018] Figure 1 This is a flowchart illustrating a method for recognizing matryoshka doll models as shown in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a single-subject bloodline knowledge graph shown in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a multi-subject kinship knowledge graph shown in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the process of constructing a bloodline knowledge graph according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the significance evaluation value of the transformation in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a matryoshka doll model recognition device shown in an embodiment of this application. Detailed Implementation

[0019] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0022] With the rapid development of big data and artificial intelligence, data platforms are widely used in enterprise-level scenarios. Users typically generate thematic data tables or models that meet business needs through SQL queries, data development scripts, or visual modeling tools.

[0023] Currently, the nested model refers to users repeatedly concatenating and trimming fields to generate new tables or models, resulting in the same entity (such as a user behavior table, order table, log table, etc.) being nested and referenced multiple times. Over time, this leads to a data model with excessively long chains and layered dependencies. For example, the nested model can cause the following problems: First, data redundancy: the same entity's fields are stored repeatedly, wasting storage space; second, lengthened chains: dependency chains become complex, requiring a single query or calculation to traverse multiple intermediate tables, increasing latency; third, maintenance difficulties: the number of table structures expands, making it difficult for developers to trace the origin of tables; fourth, degraded data quality: due to multi-layered concatenation, inconsistencies in fields and definitions can easily occur.

[0024] However, big data platforms typically provide functions such as task orchestration (DAG), metadata management, and lineage analysis. These functions mainly address task execution order (dependency management) and field lineage tracing (data traceability), but they cannot detect duplicate splicing issues in nested models.

[0025] Among related technologies, there is a lack of automated and accurate means for identifying data subjects, and difficulties in identifying the business boundaries of nested models in multi-subject mixed scenarios.

[0026] To address the aforementioned issues, this application provides a method for recognizing nested doll models, which enables automated and accurate identification of data subjects.

[0027] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating a method for recognizing nested doll models as shown in an embodiment of this application. See Figure 1 The method for recognizing a nesting doll model includes the following steps S101 to S106.

[0029] S101. Obtain business data for the business scenario and extract data subject features from the business data. The business data includes business semantic description data and data table features of the business data table.

[0030] It should be noted that data subject characteristics represent key information about the core data carrier (such as users, orders, and products), including subject identifiers, the business domain to which the subject belongs, and table-subject mapping relationships. Business semantic description data refers to metadata related to business data tables, such as table names, field comments, business subject domain divisions, and table naming conventions. Data table characteristics refer to the structure and data attributes of the data table, including field cardinality, null value rate, field relationships, primary key / foreign key information, and data sampling results.

[0031] For example, for metadata-driven identification, business semantic description data is parsed, and data subjects are initially identified and labeled based on table naming conventions, field comments, and subject domain division. For content-driven inference, in scenarios with incomplete metadata, business data tables are sampled and analyzed to identify fields with high cardinality, low null value rate, and frequent foreign key relationships (such as user_id, order_id, device_id). These fields are then determined as subject identifiers, and tables are clustered under the corresponding subjects based on these identifiers. Thus, data subject characteristics are obtained, including the subject's unique identifier, subject identifier, table-subject mapping relationship, and the business domain to which the subject belongs.

[0032] Therefore, by automatically and accurately identifying data subjects and recognizing the business boundaries of nested models in multi-subject mixed scenarios, this application enables precise location of core subjects when metadata is incomplete, achieves business context analysis, and ensures that the identification results align with actual business needs. Thus, this application improves subject identification accuracy, supports automated clustering of tables to corresponding subjects, and avoids the subjectivity and inefficiency of manual labeling.

[0033] S102. Obtain the data processing scripts related to the business scenario.

[0034] It should be noted that data processing scripts refer to the code carriers in which users execute data processing on big data platforms, including SQL queries, ETL task scripts, and underlying SQL exported by visual modeling tools. For example, based on a business scenario identifier, all data processing scripts associated with that scenario are filtered out, excluding irrelevant scripts to reduce computational overhead. The business scenario identifier can be scenario information such as BI reports, algorithm feature engineering, or layered data warehouse construction. Then, comments, line breaks, and extra spaces in the scripts are removed, and missing syntax symbols such as semicolons are added to standardize the script format. Furthermore, for dynamic SQL containing variables, pre-compilation replaces the variables with actual values ​​to ensure the completeness of subsequent parsing. From this, a script-scenario association table can be obtained, recording the business scenario to which each script belongs.

[0035] S103. Based on the data subject characteristics and the business data, the data processing script is parsed to obtain a syntax tree, and the syntax tree is traversed to construct a lineage knowledge graph. The lineage knowledge graph is used to indicate the dependencies and processing logic between graph nodes. The graph nodes include data entity nodes and / or process logic nodes.

[0036] In this application, an Abstract Syntax Tree (AST) refers to a structured hierarchical representation of a data processing script. It abstracts script keywords, tables / fields, function calls, etc., into nodes, forming a parent-child hierarchical relationship according to syntactic logic to extract dependencies. For example, using the ANTLR tool, the data processing script is transformed into a structured, hierarchical AST based on the syntax rule file, clearly defining the script's syntactic hierarchy and core elements. Then, a depth-first traversal is initiated to recursively traverse the AST. During the traversal, by accessing different types of subtree nodes such as SELECT, FROM, JOIN, and WHERE, the output target of the query (such as the table specified by INSERT INTO or CREATE TABLE AS), all input source tables, and their aliases are accurately identified.

[0037] Furthermore, by traversing through the data, column-level lineage can be traced in a penetrating manner. This allows for the parsing of every item in the SELECT list, whether it is a direct field reference, a field with an alias, a wildcard (expanded in real time by querying the metadata storage), or a complex function call (such as CASE WHEN..., window functions, UDFs) and arithmetic expressions. It can recursively establish a clear dependency relationship with one or more of the original input columns, with transformation functions and logical descriptions, thereby extracting column-level lineage.

[0038] Furthermore, physical assets (databases, tables, columns), process entities (ETL tasks, SQL queries), and processing logic (JOIN operations, function calls) are abstracted as nodes, and a multi-dimensional lineage knowledge graph (i.e., lineage knowledge graph) is constructed with dependencies / execution relationships as edges. The lineage knowledge graph refers to a semantic network constructed in the form of "nodes-edges," where nodes encompass data entity nodes (physical assets such as databases, tables, and columns) and process logic nodes (ETL tasks, JOIN operations, function calls, etc.), and edges represent the dependencies and processing logic between nodes.

[0039] Therefore, this application constructs a lineage knowledge graph to trace data dependencies, enabling fine-grained transformation analysis of the nested doll model.

[0040] S104. Mark the data subjects in the bloodline knowledge graph so that each graph node carries the subject tracing information of the corresponding data subject.

[0041] For example, based on the lineage knowledge graph and data subject characteristics (such as table-subject mapping relationships and subject unique identifiers), starting from the source table corresponding to the subject, and based on the table-subject mapping relationship, the source table and its field nodes are marked with subject unique identifiers, establishing coloring anchors to achieve source node coloring. Then, subject identifiers are passed downstream along the dependency edges of the lineage knowledge graph. Single-subject derived tables directly inherit upstream subject identifiers, while multi-subject concatenated tables (such as user tables + product tables) integrate all participating subject identifiers, ensuring that the nodes of the lineage knowledge graph carry subject traceability information. It should be noted that subject traceability information refers to the set of subject association information bound to the graph nodes of the lineage knowledge graph, including subject unique identifiers, attribution type (single subject, multi-subject), and attribution source (inheritance, concatenation, metadata identification), used to clarify the business subject attribution and evolution logic of the nodes.

[0042] Therefore, this application endows the lineage knowledge graph with business context by subject coloring, making the business subject affiliation of each graph node traceable, avoiding misjudgment by nested models, supporting accurate fusion labeling of multi-subject scenarios, adapting to the needs of complex business data splicing, and improving recognition accuracy and efficiency.

[0043] In some embodiments, the subject tracing information is used to indicate the subject identifier of at least one data subject corresponding to the upstream graph node of each graph node, and there is a calling relationship between the at least one data subject and the data subject corresponding to the graph node.

[0044] In this embodiment, the subject tracing information mainly refers to the data subject attribution and formation logic of each graph node, including the unique identifier of one or more subjects corresponding to the node, the attribution type (single subject, multiple subjects), and the source of attribution (such as inheriting from upstream subjects, splicing multiple upstream subjects, or direct identification of metadata). Therefore, this embodiment can achieve automated and accurate identification of data subjects, identify the business boundaries of nested models in multi-subject mixed scenarios, establish subject-link associations, and enable nested model identification to associate with business context anchor points.

[0045] For example, Figure 2 This is a schematic diagram of the structure of a single-subject lineage knowledge graph shown in the embodiments of this application. All nodes are derived from DB1.tab_01 and are continuous single-table dependency links. Starting from the source table representing the subject (e.g., ods_user_log), all downstream tables directly derived from it will be marked with the unique identifier of the subject in the lineage graph.

[0046] For example, Figure 3 This is a schematic diagram illustrating the structure of a multi-subject lineage knowledge graph according to an embodiment of this application. A downstream wide table may be composed of data from multiple data subjects (for example, a user behavior wide table may contain information from both user and product subjects). In this case, the model and its fields are marked with subject identifiers from multiple data subjects, enabling multi-subject transfer.

[0047] S105. Based on the labeled kinship knowledge graph and the syntax tree, determine the transformation significance evaluation value, which is used to reflect the degree of transformation of each data transformation.

[0048] It should be noted that the transformation significance evaluation value refers to a comprehensive score that quantifies the degree of a single data transformation. The higher the score, the more significant the transformation. It is obtained by weighted fusion of three dimensions: structural change degree, logical complexity, and correlation breadth. Therefore, this application realizes the transformation of abstract data transformation into a quantifiable score, improves the accuracy of identifying high-inertia transformations, and can distinguish between low-value redundant transformations and high-value effective transformations, thereby improving the accuracy of evaluation value calculation.

[0049] S106. Determine the recognition result of the matryoshka model based on the labeled bloodline knowledge graph and the transformation significance evaluation value.

[0050] It should be noted that the "nested doll model" refers to a data model with excessively long and layered dependencies formed by multiple low-value nested transformations (splicing, pruning, etc.) of the same main data. Its characteristic is a long chain composed of continuous high-inertia transformations. For example, data transformation operations with a significance evaluation value below a significance threshold are marked as high-inertia transformation operations, while the rest are marked as valid transformations. Then, a hybrid graph algorithm combining depth-first and breadth-first search is used to search the graph for long chains composed of continuous high-inertia transformations, while simultaneously verifying the consistency of the main data within the chain. Furthermore, chains that satisfy the conditions of having more than a preset threshold of continuous high-inertia transformations, having a main data consistency greater than a preset threshold, and having no valid transformations inserted are determined to be nested doll models.

[0051] Optionally, output a list of nested model identification results, including link ID, risk level, node sequence, subject information, transformation significance assessment value, and judgment basis. The risk classification visualization report includes displaying the distribution of nested models, link length distribution, subject distribution, and governance priority ranking table, that is, sorted by risk level, data volume, and business impact scope, and clearly identifying priority governance targets.

[0052] Therefore, this application can accurately locate nested models in big data platforms, improve the accuracy of nested model identification, avoid omissions or misjudgments, distinguish nested models of different risk levels, optimize the allocation of governance resources, and improve governance efficiency.

[0053] In some embodiments, the data subject features include the data subject and the subject identifier and mapping relationship data corresponding to the data subject, wherein the mapping relationship data is used to indicate the association between the business data table and the data subject. Figure 4 This is a schematic diagram of the process of constructing a kinship knowledge graph according to an embodiment of this application. The step of parsing the data processing script based on the data subject characteristics and the business data to obtain a syntax tree, and traversing the syntax tree to construct a kinship knowledge graph includes the following steps S401 to S404.

[0054] S401. Using a preset parser generator, the data processing script is converted into a syntax tree; S402. Recursively traverse the syntax tree and extract the lineage relationship tuples; S403. Based on the lineage relationship tuple, construct a column-level lineage relationship diagram, which is used to indicate the source information, destination information and data subject to which each field in the business data table belongs; S404. Construct a bloodline knowledge graph based on the column-level bloodline relationship diagram.

[0055] It should be noted that a data subject refers to the analytical object in data modeling (such as a user, order, product, or device), and a subject identifier is a unique identifier for each data subject, used to clearly distinguish different subjects in the graph. Mapping relationship data refers to structured data that records the relationships between business data tables and data subjects, supporting precise binding between fields and subjects.

[0056] In this embodiment, the parser generator refers to ANTLR (Another Tool for Language Recognition), an open-source syntax parsing tool that transforms data processing scripts into syntax trees based on custom SQL syntax rules. This syntax tree contains a hierarchical node structure representing the complete syntactic logic of the script. Each node carries basic information such as token type, related tables / fields, etc., thus transforming unstructured data processing scripts into a machine-readable structured carrier. Next, a recursive traverser is initialized, employing a depth-first traversal algorithm. Starting from the root node of the syntax tree, it traverses the child nodes such as SELECT, FROM, JOIN, and WHERE layer by layer. This process extracts structured lineage information units (i.e., lineage relation tuples) from the syntax tree. For example, these lineage relation tuples contain complete lineage information for all fields. Each tuple contains key information such as output fields, input fields, processing logic, the subject, and subject identifier, enabling the identification of business context and technical dependencies.

[0057] Furthermore, nodes are defined, abstracting each field as a field node, containing attributes such as field name, table, data type, subject identifier, and field comments. Edges are defined, treating the dependencies in the lineage tuples as lineage edges, containing attributes such as processing logic and data flow direction (input → output). Therefore, following the logic of "field node → lineage edge → field node," all lineage tuples are transformed into a visual graph structure, clearly defining the upstream source (input field), downstream destination (output field), and subject of each field, thus constructing a column-level lineage graph. This column-level lineage graph indicates the upstream source, downstream destination, and data subject to which each field belongs, enabling field-level traceability and field dependencies, thereby intuitively presenting the entire link flow trajectory of fields and their affiliation with the business subject.

[0058] Furthermore, for example, based on the column-level lineage graph, three new types of nodes are added: First, physical asset nodes: databases, schemas, tables (extended based on the table hierarchy to which a field belongs); second, process entity nodes: ETL tasks, SQL queries (processing scripts for related fields); third, processing logic nodes: JOIN operations, function calls (based on the abstract processing logic in tuples). Edge expansion: First, attribution edge: database → schema → table → column (reflecting hierarchical attribution); second, processing edge: ETL task → table / field (reflecting the processing relationship between task and data); third, logical association edge: processing logic node → field (reflecting the association between logic and field); fourth, subject association edge: subject identifier → table / field (reflecting the binding relationship between subject and data). Thus, the expanded nodes and edges are integrated into a unified semantic network, resulting in a lineage knowledge graph.

[0059] Therefore, this embodiment achieves deep integration of lineage and subject. Through full-process subject binding, each data node (table / field), process node (task / script), and logic node (function / JOIN) carries clear business subject information, and the data semantics can be directly traced, enabling automated and accurate identification of the data subject.

[0060] In some embodiments, determining the transformation significance evaluation value based on the labeled kinship knowledge graph and the syntax tree includes: Based on the labeled lineage knowledge graph and the syntax tree, the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value are determined. The structural change degree evaluation value is used to reflect the degree of change of the field set, the logical complexity evaluation value is used to reflect the complexity of the data processing logic, and the association breadth evaluation value is used to reflect the association complexity of the data transformation. Assign a first weight to the structural change degree evaluation value, a second weight to the logical complexity evaluation value, and a third weight to the association breadth evaluation value; Based on the first weight, the second weight, and the third weight, the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value are weighted and fused to obtain the transformation significance evaluation value.

[0061] It should be noted that the transformation significance assessment value is a comprehensive score that quantifies the degree of substantial change in a single data transformation. A higher score indicates a more significant transformation (higher information gain), used to identify high-inertia transformation operations. Specifically, the structural change assessment value is a normalized score reflecting the degree of change in the set of fields before and after the data transformation, measuring the scale of field additions, deletions, and modifications, i.e., the number and proportion of fields added, deleted, or modified in the SELECT list. The logical complexity assessment value is a normalized score reflecting the complexity of the data processing logic, focusing on the logical difficulty of function calls, predicate nesting, etc., i.e., the number and nesting depth of predicates in the WHERE / HAVING clauses, and the use of aggregate functions, window functions, and UDFs. The association breadth assessment value is a normalized score reflecting the complexity of data transformation associations, measuring the number and type complexity of JOIN operations, i.e., the number of JOIN operations and the number of associated tables.

[0062] In this embodiment, the first weight refers to the weight coefficient assigned to the structural change degree assessment value, used to adjust the contribution ratio of structural changes to the total score. The second weight refers to the weight coefficient assigned to the logical complexity assessment value, used to adjust the contribution ratio of logical difficulty to the total score. The third weight refers to the weight coefficient assigned to the association breadth assessment value, used to adjust the contribution ratio of association complexity to the total score.

[0063] For example, the significance evaluation value of the transformation is calculated using the following formula:

[0064] in, This represents the significance evaluation value of the transformation. This represents the structural change assessment value. This represents the logical complexity evaluation value. This represents the assessment value of the breadth of association. Indicates the first weight. Indicates the second weight. This indicates the third weight.

[0065] For example, Figure 5 This is a schematic diagram illustrating the transformation significance evaluation value in an embodiment of this application. For node DB1.tab_03, whose data body belongs to nodes DB1.tab_01 and DB1.tab_02, the transformation significance evaluation value is 0.11 for the data body from node DB1.tab_01 (body identifier is 7278236782436352) and 0.82 for the data body from node DB1.tab_02 (body identifier is 7278236782436353).

[0066] Therefore, this embodiment weights and fuses the three-dimensional evaluation values ​​with their corresponding weights to determine the significance evaluation value of the transformation, thereby realizing the evaluation of data transformation and achieving multi-dimensional accurate quantitative evaluation.

[0067] In some embodiments, the values ​​of the first weight, the second weight, and the third weight are associated with the scenario profile of the business scenario.

[0068] In this embodiment, a dynamic adjustment mechanism that intelligently correlates with business scenarios is introduced. The transformation significance evaluation value takes into account the deep correlation with business logic and application scenarios, as different business scenarios have different focuses on data processing. For example, a report model oriented towards a management dashboard focuses primarily on the accurate aggregation of multi-source data and the calculation of complex business rules; while a dataset preparation task oriented towards algorithm training focuses primarily on the effective screening and derivation of original features.

[0069] Therefore, this embodiment determines the weight values ​​for the structural change assessment value, logical complexity assessment value, and association breadth assessment value based on the scenario profile of the business scenario. When performing nested model recognition, the data link to be analyzed is associated with the corresponding scenario profile, thereby using the most suitable weight combination to calculate the change significance assessment value.

[0070] For example, let's take BI reports and data visualization as an example. This scenario highly values ​​the consistency of the final data's business scope, the accuracy of the calculation logic, and the ability to integrate multiple business themes. Data processing typically involves a large amount of aggregation, window functions, and associations with dimension tables. Therefore, the second weight is the largest, as the calculation logic of indicators in reports is usually very complex (year-on-year, month-on-month, percentage, etc.), hence the weight for logical complexity is the largest. The third weight is greater than the first weight. Reports need to associate multiple dimension tables to enrich data dimensions, and the breadth of associations is also important. Simple field additions and deletions are usually not very valuable in this scenario, unless generating key derived indicators. For example, the second weight is 0.5, the third weight is 0.3, and the first weight is 0.2.

[0071] For example, let's take algorithmic feature engineering as an example. The goal of this scenario is to extract, filter, and create features beneficial to model training from raw data. The focus of data processing is on the transformation, derivation, and filtering of fields. Therefore, the first weight should be the largest, as the addition, deletion, and modification of fields directly correspond to the creation and selection of features, measuring the value of the transformation, and should therefore have the highest weight. The second weight should be greater than the third weight. Feature generation often involves certain computational logic (such as binning, normalization, etc.), and logical complexity should also account for a certain proportion. Feature engineering is sometimes a single-table operation, and although it may be related, its importance is usually lower than the construction of the feature itself. For example, the first weight is 0.6, the second weight is 0.3, and the third weight is 0.1.

[0072] For example, taking the layered construction of a data warehouse as an illustration, the processing objectives differ at different levels when building a standard data warehouse chain. For instance, from the DWD to the DWS layer, the focus is on integrating multiple detailed data sets to form a wide table representing a business theme. Therefore, the third weight has the highest value. The main operation at this stage is to join and concatenate the fact table with multiple dimension tables, so the weight of the breadth of association should be the highest. The first weight is greater than the second weight; the number of fields usually increases significantly, and the structural changes are also more pronounced. The DWS layer may only involve light aggregation or logical calculations, leaving more complex logic to the ADS layer. For example, the third weight is 0.5, the first weight is 0.3, and the second weight is 0.2.

[0073] Therefore, this embodiment can discover inefficient data links from a purely technical perspective, and can also accurately identify redundant transformations with low value in specific business scenarios by combining business context, which greatly improves the intelligence level and accuracy of automated data governance.

[0074] In some embodiments, the structural change assessment value is determined by the following steps: Based on the list of source tables extracted from the FROM clause of the syntax tree, determine the total number of input fields involved in the data transformation; The final number of output fields in the SELECT list is determined based on the output field items extracted from the SELECT list in the syntax tree; The number of new fields is determined based on the column-level lineage diagram, the SELECT list of the syntax tree, and the number of output fields. The number of fields to be modified is determined based on the syntax tree, the number of newly added fields, the number of output fields, and the column-level lineage diagram. The number of fields to be passed through is determined based on the number of output fields, the number of newly added fields, and the number of modified fields. Based on the column-level lineage diagram, determine the input fields referenced by the output, which will be the number of input fields used. The number of fields to be deleted is determined based on the total number of input fields and the number of input fields used. The total number of fields that have changed is determined based on the number of newly added fields, the number of modified fields, and the number of deleted fields; Based on the total number of input fields and the number of newly added fields, determine the total field space involved in the data transformation; The structural change assessment value is determined based on the total number of fields that have changed and the total field space.

[0075] It should be noted that the FROM clause of the syntax tree contains information about the source tables in direct references, JOINed tables, and nested subqueries. The AST nodes of the FROM, JOIN, and nested subqueries in the syntax tree are traversed to collect all source tables involved in the transformation. Then, through metadata storage (i.e., the field list of the source tables and table-field mapping relationships), the data is expanded to include all fields of the source tables. Based on the column-level lineage graph (the hierarchical dependencies of the source tables), all fields from the source tables are merged, duplicate fields are removed, and the total number of input fields is output. The total number of input fields refers to the total number of duplicate fields from all source tables involved in the data transformation (including directly referenced tables and nested subquery source tables), measuring the scale of the input data.

[0076] Next, the child nodes of the SELECT list in the syntax tree are traversed, collecting all output elements (including fields with aliases, function results, and constants), i.e., extracting output field items, which are then expanded into specific fields through metadata storage. Then, based on the column-level lineage graph (the hierarchical dependencies of the source table), completely duplicate output fields (field names and dependencies are identical) are excluded, resulting in the number of output fields. This number of output fields refers to the total number of fields finally output in the SELECT list of the syntax tree (including aliases, after deduplication), reflecting the scale of the transformed data format.

[0077] Furthermore, based on the input dependencies of the output fields in the column-level lineage graph, fields marked "no input dependency" are filtered from the output fields. The newly added attributes are verified based on the number of output fields and the SELECT list of the syntax tree (the generation logic of the output fields), and duplicate new fields (with identical field names and generation logic) are excluded, thus obtaining the number of new fields. This number of new fields refers to the number of fields without input dependencies, created by constants, independent expressions, or parameterless functions.

[0078] Furthermore, based on the number of newly added fields, the number of output fields, and the function call nodes, expression nodes, and type conversion nodes in the SELECT list of the syntax tree, the number of newly added fields is excluded from the output fields, and fields marked with "input dependency" at the column level are retained. Modified attributes are then verified to obtain the number of modified fields. This number of modified fields refers to the number of fields that depend on input fields but have undergone function, expression, or type conversion processing, and are not passed through as is.

[0079] Furthermore, the mutual exclusion relationship of field types is shown in the following formula:

[0080] in, Indicates the number of output fields. Indicates the number of newly added fields. Indicates the number of fields to be modified. This indicates the number of fields passed through. Based on the mutual exclusion relationship of field types, the number of fields passed through is determined by the number of output fields, the number of newly added fields, and the number of modified fields. This number of fields passed through refers to the number of fields directly passed from the input fields to the output without any modification (including those only renamed).

[0081] Next, the column-level lineage graph is traversed to collect all input fields referenced by output fields (including directly referenced fields and fields involved in function / expression evaluations) to extract referencing fields and determine the number of used input fields. The number of used input fields refers to the total number of duplicate input fields referenced by output fields (including directly referenced fields and fields involved in function / expression evaluations). Therefore, based on the total number of input fields and the number of used input fields, the number of deleted fields is calculated. This number refers to the number of fields that exist in the total number of input fields but are not referenced by the output or involved in any output field evaluations.

[0082] Then, the structural change assessment value is calculated using the following formula:

[0083] in, Indicates the number of fields deleted. This represents the total number of input fields. The numerator of this formula is determined by the number of newly added fields, modified fields, and deleted fields, which is the total number of fields that have changed. This total number of changed fields refers to the sum of the number of newly added, modified, and deleted fields, reflecting the absolute scale of the structural change. The denominator of this formula is determined by the number of newly added fields and the total number of input fields, which is the total field space. This total field space refers to the entire range of fields involved in the data transformation, i.e., the sum of the total number of input fields and the number of newly added fields, and is used for normalization calculations.

[0084] Therefore, this embodiment transforms the abstract field addition, deletion and modification situation into a quantifiable normalized score, realizing accurate quantification of structural changes, which can cover complex scenarios and thus improve the accuracy of high inertia transformation judgment.

[0085] In some embodiments, the logical complexity evaluation value is determined by the following steps: Traverse all function calls in the syntax tree, sum the called functions by weight, and obtain the function complexity score; Extract the corresponding logical structure based on the WHERE and HAVING clauses of the syntax tree; Based on the logical structure and the preset predicate recognition rules, determine the total number of predicates and the maximum nesting depth of logical operators; The predicate complexity score is determined based on the total number of predicates and the maximum nesting depth. The logical complexity evaluation value is determined based on the function complexity score and the predicate complexity score.

[0086] It should be noted that traversing function call nodes involves initiating a depth-first search traversal, recursively traversing all function call nodes and expression nodes in the syntax tree, and collecting all functions and operations involved in data processing. Next, function type matching and weight assignment occur, matching according to function type labeling rules and assigning corresponding weights to each function and operation. If function parameters contain nested functions, nested functions are recursively identified and their weights accumulated. Finally, the weights of all functions / operations are summed to obtain a function complexity score. This function complexity score is obtained by weighted summation of all functions / operations in the syntax tree according to preset weights, reflecting the computational depth of the data processing logic. The weighted summation of nested functions quantifies the computational depth and logical difficulty of function operations, distinguishing between basic and complex operations.

[0087] Next, the WHERE and HAVING child nodes are precisely located in the syntax tree. The target child nodes are traversed, and all constituent elements (atomic predicates, logical operators AND / OR, and parentheses) are extracted to obtain the logical structure elements. These elements are then abstracted into nodes (predicate nodes, logical operator nodes, and parenthesis nodes) according to the syntactic hierarchy, forming a hierarchical logical structure tree, i.e., the logical structure. This logical structure refers to the hierarchical syntactic structure in the WHERE / HAVING clause, composed of predicates, logical operators (AND / OR), and parentheses.

[0088] Furthermore, to count the total number of predicates, the logical structure tree is traversed, and atomic predicate nodes (indivisible filtering units) are filtered according to the predicate recognition rules. Logical operator nodes and parenthesis nodes are excluded. The atomic predicates are deduplicated and counted; the count result is the total number of predicates. Here, the total number of predicates refers to the deduplicated count of all atomic predicates in the WHERE / HAVING clause, reflecting the scale of the filtering rules. To calculate the maximum nesting depth, the parentheses in the logical structure tree are mapped to level markers. All logical operator nodes are traversed, and the level of each operator is recorded. The maximum value of all levels is taken as the maximum nesting depth of the logical operator. Here, the maximum nesting depth of the logical operator refers to the highest level of the logical operator in the parenthesis nesting (level 0 for no nesting, level 1 for the innermost parenthesis), reflecting the nesting complexity of the filtering logic. Predicate recognition rules refer to predefined atomic predicate judgment criteria (such as combinations of "field + comparison operator + value / field") used to distinguish predicates from non-predicate elements such as logical operators and parentheses.

[0089] Furthermore, the predicate complexity score is calculated using the following formula:

[0090] in, This represents the predicate complexity score. Indicates the total number of predicates. This represents the maximum nesting depth of logical operators. The predicate complexity score quantifies the logical complexity of the WHERE / HAVING clause filtering logic, and is derived from the total number of predicates and the maximum nesting depth of logical operators.

[0091] Then, the logical complexity evaluation value is calculated using the following formula:

[0092] in, Indicates the calibration coefficient. This represents the function complexity score. Therefore, this embodiment takes into account the depth of function operations and the complexity of the filtering logic to improve the accuracy of the evaluation.

[0093] In some embodiments, the association breadth evaluation value is determined by the following steps: Based on the column-level lineage diagram and the FROM and JOIN clauses of the syntax tree, determine the total number of tables involved in the data transformation; The association complexity score is determined based on the column-level lineage graph, the JOIN clauses of the syntax tree, the total number of tables, and the preset JOIN operation weight dictionary. The association breadth evaluation value is determined based on the association complexity score.

[0094] In this embodiment, a depth-first traversal is initiated, recursively traversing the AST nodes of the FROM clause, JOIN clause, and nested subqueries in the syntax tree. All source tables participating in the transformation are collected, and duplicate table names are removed to avoid repeated counting due to multiple references to the same table. Invalid tables are also excluded. The total number of valid tables after deduplication is the total number of tables participating in the data transformation. This total number of tables participating in the data transformation refers to the total number of all source tables involved in this data transformation (including directly referenced tables and source tables in nested subqueries).

[0095] Furthermore, traverse the JOIN clauses of the syntax tree, extract all valid JOIN operations, form a list of JOIN operations, and perform a JOIN operation on all ( The association complexity score is obtained by weighted summation of JOIN operations. For example, the association complexity score can be calculated using the following formula:

[0096] in, Indicates the correlation complexity score. This indicates the total number of tables involved in the data transformation. This represents the dictionary of weights for the JOIN operation. This represents the j-th JOIN operation. The association complexity score is a score obtained by weighting all JOIN operations according to preset weights, reflecting the overall complexity of the association operations. The JOIN operation weight dictionary is a pre-configured mapping table of complexity weights for different JOIN types (e.g., inner join weight 1.0, cross join weight 3.0). The more complex the logic and the higher the risk of data expansion, the higher the weight.

[0097] Furthermore, the association breadth evaluation value is calculated using the following formula:

[0098] in, This represents the calibration coefficient. The formula reflects that the more tables involved and the more complex the association method, the higher the score, approaching 1. This is a calibration coefficient (e.g., k=0.5) used to adjust the sensitivity of the contribution of association breadth to the total score. It can be adjusted according to platform practice and should be consistent with the calibration coefficient in the calculation of the logical complexity evaluation value. Therefore, this embodiment can improve the accuracy of long-link determination in the nested model, thus effectively improving the recognition accuracy of the nested model.

[0099] In some embodiments, determining the Matryoshka doll model recognition result based on the labeled kinship knowledge graph and the transformation significance evaluation value includes: Traverse the marked kinship knowledge graph and mark data transformation operations with transformation significance evaluation values ​​lower than a preset significance threshold as high inertia transformation operations; In the labeled lineage knowledge graph, a long link consisting of multiple consecutive high-inertia transformation operations is searched, and the long link is identified as a nested model.

[0100] In this embodiment, a high-inertia transformation operation refers to a single data processing operation where the significance evaluation value of the data transformation is lower than a preset significance threshold. Its characteristics include extremely low degree of structural and logical modification of the data, and the output data has a highly similar information value to the input data (such as simple field trimming or renaming). The significance threshold is used to distinguish between high-inertia transformations and effective transformations.

[0101] For example, the pedigree knowledge graph after marking is traversed, and for each transformed edge, the transformation saliency evaluation value is extracted. If the transformation saliency evaluation value is lower than the saliency threshold, the transformation operation is marked as a high-inertia transformation operation, and its transformation saliency evaluation value, the subject to which it belongs, and the upstream and downstream node information are recorded. If the transformation saliency evaluation value is not lower than the saliency threshold, it is marked as a valid transformation operation and excluded from the subsequent long-link search range.

[0102] Furthermore, for example, during the search link process, link validity checks are performed. These checks include: Consecutive count check: ensuring the number of consecutive high-inertia transformation operations in the link exceeds a preset consecutive count threshold; Subject consistency check: calculating the proportion of the same subject in the link using subject tracing information from graph nodes, which must be greater than or equal to a subject consistency threshold (e.g., 80%) to exclude cross-subject, disorderly links; No valid transformation check: ensuring there are no valid transformation operations in the link. Therefore, long links that satisfy all the above link validity checks are classified as nested models.

[0103] Therefore, this embodiment identifies a long link consisting of multiple consecutive high-inertia transformation operations as a nested model, thereby achieving automated and accurate identification of the nested model and effectively improving the accuracy and efficiency of nested model identification.

[0104] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a matryoshka doll model recognition device and corresponding embodiments.

[0105] Figure 6 This is a schematic diagram of the structure of a matryoshka doll model recognition device shown in an embodiment of this application.

[0106] See Figure 6 The matryoshka doll model recognition device 600 includes: The business data acquisition module 601 is used to acquire business data of a business scenario and extract data subject features from the business data. The business data includes business semantic description data of the business data table and data table features. Script acquisition module 602 is used to acquire data processing scripts related to the business scenario; The knowledge graph construction module 603 is used to parse the data processing script based on the data subject characteristics and the business data to obtain a syntax tree, and traverse the syntax tree to construct a lineage knowledge graph. The lineage knowledge graph is used to indicate the dependency relationships and processing logic between graph nodes. The graph nodes include data entity nodes and / or process logic nodes. The subject marking module 604 is used to mark the data subjects in the bloodline knowledge graph, so that each graph node carries the subject tracing information of the corresponding data subject; The transformation significance evaluation module 605 is used to determine the transformation significance evaluation value based on the labeled kinship knowledge graph and the syntax tree. The transformation significance evaluation value is used to reflect the degree of transformation of each data transformation. The identification module 606 is used to determine the identification result of the matryoshka model based on the labeled bloodline knowledge graph and the transformation significance evaluation value.

[0107] In some embodiments, the data subject features include a data subject and a subject identifier and mapping relationship data corresponding to the data subject, wherein the mapping relationship data is used to indicate the association between the business data table and the data subject; The knowledge graph construction module 603 includes: A syntax tree building unit is used to convert the data processing script into a syntax tree using a preset parser generator; The bloodline relationship tuple extraction unit is used to recursively traverse the syntax tree and extract the bloodline relationship tuples. The column-level lineage graph construction unit is used to construct a column-level lineage graph based on the lineage tuples. The column-level lineage graph is used to indicate the source information, destination information and data body to which each field in the business data table belongs. The bloodline knowledge graph generation unit is used to construct a bloodline knowledge graph based on the column-level bloodline relationship graph.

[0108] In some embodiments, the subject tracing information is used to indicate the subject identifier of at least one data subject corresponding to the upstream graph node of each graph node, and there is a calling relationship between the at least one data subject and the data subject corresponding to the graph node.

[0109] In some embodiments, the transformation significance evaluation module 605 includes: The calculation unit is used to determine the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value based on the labeled lineage knowledge graph and the syntax tree. The structural change degree evaluation value is used to reflect the degree of change of the field set, the logical complexity evaluation value is used to reflect the complexity of the data processing logic, and the association breadth evaluation value is used to reflect the association complexity of the data transformation. A weight allocation unit is used to allocate a first weight to the structural change degree evaluation value, a second weight to the logical complexity evaluation value, and a third weight to the association breadth evaluation value; The weighted fusion processing unit is used to perform weighted fusion processing on the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value according to the first weight, the second weight, and the third weight to obtain the transformation significance evaluation value.

[0110] In some embodiments, the computing unit includes: The input total field count calculation unit is used to determine the total number of input fields participating in data transformation based on the list of source tables extracted from the FROM clause of the syntax tree. The output field count calculation unit is used to determine the final number of output fields in the SELECT list based on the output field items extracted from the SELECT list in the syntax tree. The newly added field count calculation unit is used to determine the number of new fields based on the column-level lineage diagram, the SELECT list of the syntax tree, and the number of output fields. The modified field quantity calculation unit is used to determine the number of modified fields based on the syntax tree, the number of newly added fields, the number of output fields, and the column-level lineage diagram; The transparent field quantity calculation unit is used to determine the number of transparent fields based on the number of output fields, the number of newly added fields, and the number of modified fields; The input field count calculation unit is used to determine the input fields referenced by the output based on the column-level lineage diagram, which is the number of input fields used. The field deletion count calculation unit is used to determine the number of fields to be deleted based on the total number of input fields and the number of input fields used. The total number of fields that have changed is calculated by the unit for determining the total number of fields that have changed based on the number of newly added fields, the number of modified fields, and the number of deleted fields. The total field space calculation unit is used to determine the total field space involved in the data transformation based on the total number of input fields and the number of newly added fields. The structural change degree assessment value calculation unit is used to determine the structural change degree assessment value based on the total number of fields that have changed and the total field space.

[0111] In some embodiments, the computing unit includes: The function complexity score calculation unit is used to traverse all function calls in the syntax tree, perform a weighted summation of the called functions, and obtain the function complexity score. The logical structure extraction unit is used to extract the corresponding logical structure based on the WHERE clause and HAVING clause of the syntax tree; The predicate analysis unit is used to determine the total number of predicates and the maximum nesting depth of logical operators based on the logical structure and preset predicate recognition rules. A predicate complexity scoring unit is used to determine a predicate complexity score based on the total number of predicates and the maximum nesting depth. The logic complexity evaluation unit is used to determine the logic complexity evaluation value based on the function complexity score and the predicate complexity score.

[0112] In some embodiments, the computing unit includes: The total number of tables is calculated based on the column-level lineage diagram and the FROM and JOIN clauses of the syntax tree to determine the total number of tables involved in the data transformation. The association complexity score calculation unit is used to determine the association complexity score based on the column-level lineage graph, the JOIN clause of the syntax tree, the total number of tables, and a preset JOIN operation weight dictionary. The association breadth evaluation value calculation unit is used to determine the association breadth evaluation value based on the association complexity score.

[0113] In some embodiments, the values ​​of the first weight, the second weight, and the third weight are associated with the scenario profile of the business scenario.

[0114] In some embodiments, the identification module 606 includes: A high-inertia transformation operation marking unit is used to traverse the marked kinship knowledge graph and mark data transformation operations whose transformation significance evaluation value is lower than a preset significance threshold as high-inertia transformation operations. The Matryoshka doll model recognition unit is used to search for long links consisting of multiple consecutive high-inertia transformation operations in the marked lineage knowledge graph, and to identify the long links as Matryoshka doll models.

[0115] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0116] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for recognizing matryoshka doll models, characterized in that, include: Acquire business data from a business scenario and extract data subject features from the business data, which includes business semantic description data and data table features of the business data table; Obtain the data processing scripts related to the business scenario; Based on the data subject characteristics and the business data, the data processing script is parsed to obtain a syntax tree, and the syntax tree is traversed to construct a lineage knowledge graph. The lineage knowledge graph is used to indicate the dependencies and processing logic between graph nodes, and the graph nodes include data entity nodes and / or process logic nodes. The bloodline knowledge graph is labeled with data subjects, so that each graph node carries the subject tracing information of the corresponding data subject; Based on the labeled kinship knowledge graph and the syntax tree, a transformation significance evaluation value is determined, which is used to reflect the degree of transformation of each data transformation; The recognition result of the matryoshka doll model is determined based on the labeled kinship knowledge graph and the transformation significance evaluation value.

2. The nesting doll model recognition method according to claim 1, characterized in that, The data subject characteristics include the data subject and the subject identifier and mapping relationship data corresponding to the data subject. The mapping relationship data is used to indicate the association between the business data table and the data subject. The step of parsing the data processing script based on the data subject characteristics and the business data to obtain a syntax tree, and then traversing the syntax tree to construct a lineage knowledge graph, includes: The data processing script is converted into a syntax tree using a preset parser generator. The syntax tree is recursively traversed to extract the lineage relationship tuples; Based on the lineage relationship tuple, a column-level lineage relationship diagram is constructed. The column-level lineage relationship diagram is used to indicate the source information, destination information and the data body to which each field in the business data table belongs. Based on the column-level kinship diagram, a kinship knowledge graph is constructed.

3. The nesting doll model recognition method according to claim 2, characterized in that, The subject traceability information is used to indicate the subject identifier of at least one data subject corresponding to the upstream graph node of each graph node, and there is a calling relationship between the at least one data subject and the data subject corresponding to the graph node.

4. The nesting doll model recognition method according to claim 3, characterized in that, The step of determining the significance evaluation value of the transformation based on the labeled kinship knowledge graph and the syntax tree includes: Based on the labeled lineage knowledge graph and the syntax tree, the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value are determined. The structural change degree evaluation value is used to reflect the degree of change of the field set, the logical complexity evaluation value is used to reflect the complexity of the data processing logic, and the association breadth evaluation value is used to reflect the association complexity of the data transformation. Assign a first weight to the structural change degree evaluation value, a second weight to the logical complexity evaluation value, and a third weight to the association breadth evaluation value; Based on the first weight, the second weight, and the third weight, the structural change degree evaluation value, the logical complexity evaluation value, and the association breadth evaluation value are weighted and fused to obtain the transformation significance evaluation value.

5. The nesting doll model recognition method according to claim 4, characterized in that, The structural change assessment value is determined through the following steps: Based on the list of source tables extracted from the FROM clause of the syntax tree, determine the total number of input fields involved in the data transformation; The final number of output fields in the SELECT list is determined based on the output field items extracted from the SELECT list in the syntax tree; The number of new fields is determined based on the column-level lineage diagram, the SELECT list of the syntax tree, and the number of output fields. The number of fields to be modified is determined based on the syntax tree, the number of newly added fields, the number of output fields, and the column-level lineage diagram. The number of fields to be passed through is determined based on the number of output fields, the number of newly added fields, and the number of modified fields. Based on the column-level lineage diagram, determine the input fields referenced by the output, which will be the number of input fields used. The number of fields to be deleted is determined based on the total number of input fields and the number of input fields used. The total number of fields that have changed is determined based on the number of newly added fields, the number of modified fields, and the number of deleted fields; Based on the total number of input fields and the number of newly added fields, determine the total field space involved in the data transformation; The structural change assessment value is determined based on the total number of fields that have changed and the total field space.

6. The nesting doll model recognition method according to claim 4, characterized in that, The logical complexity evaluation value is determined through the following steps: Traverse all function calls in the syntax tree, sum the called functions by weight, and obtain the function complexity score; Extract the corresponding logical structure based on the WHERE and HAVING clauses of the syntax tree; Based on the logical structure and the preset predicate recognition rules, determine the total number of predicates and the maximum nesting depth of logical operators; The predicate complexity score is determined based on the total number of predicates and the maximum nesting depth. The logical complexity evaluation value is determined based on the function complexity score and the predicate complexity score.

7. The nesting doll model recognition method according to claim 4, characterized in that, The following steps are used to determine the association breadth assessment value, including: Based on the column-level lineage diagram and the FROM and JOIN clauses of the syntax tree, determine the total number of tables involved in the data transformation; The association complexity score is determined based on the column-level lineage graph, the JOIN clauses of the syntax tree, the total number of tables, and the preset JOIN operation weight dictionary. The association breadth evaluation value is determined based on the association complexity score.

8. The nesting doll model recognition method according to claim 4, characterized in that, The values ​​of the first weight, the second weight, and the third weight are associated with the scenario profile of the business scenario.

9. The nesting doll model recognition method according to claim 1, characterized in that, The step of determining the Matryoshka doll model recognition result based on the labeled kinship knowledge graph and the transformation significance evaluation value includes: Traverse the marked kinship knowledge graph and mark data transformation operations with transformation significance evaluation values ​​lower than a preset significance threshold as high inertia transformation operations; In the labeled lineage knowledge graph, a long link consisting of multiple consecutive high-inertia transformation operations is searched, and the long link is identified as a nested model.

10. A matryoshka doll model recognition device, characterized in that, include: The business data acquisition module is used to acquire business data of a business scenario and extract data subject features from the business data. The business data includes business semantic description data of the business data table and data table features. The script acquisition module is used to acquire data processing scripts related to the business scenario. The knowledge graph construction module is used to parse the data processing script based on the data subject characteristics and the business data to obtain a syntax tree, and traverse the syntax tree to construct a lineage knowledge graph. The lineage knowledge graph is used to indicate the dependencies and processing logic between graph nodes. The graph nodes include data entity nodes and / or process logic nodes. The subject tagging module is used to tag the data subjects in the bloodline knowledge graph, so that each graph node carries the subject tracing information of the corresponding data subject; The transformation significance evaluation module is used to determine the transformation significance evaluation value based on the labeled kinship knowledge graph and the syntax tree. The transformation significance evaluation value is used to reflect the degree of transformation of each data transformation. The identification module is used to determine the identification result of the matryoshka model based on the labeled kinship knowledge graph and the transformation significance evaluation value.

Images