Automated metadata management and apparatus for data knitting management

Abstract

The invention provides a device for automatic metadata management and data knitting treatment, and the device comprises a semantic mounting unit which is used for carrying out the semantic tag mounting of a structured metadata set of a data factory based on a semantic model of a data knitting architecture, so as to generate enhanced semantic metadata and a complete data line record; the rule binding unit is used for carrying out governance rule binding on the enhanced semantic metadata and the complete data line record on the basis of a data knitting federation governance rule so as to generate a metadata governance list bound with the governance rule; the change updating unit is used for synchronously updating a data set state field and a flow path recorded by a corresponding complete data line to a metadata governance list based on a change data capture mechanism when it is monitored that a data factory has a data set change, so as to generate unified metadata caalog; the interface configuration unit is used for configuring a metadata retrieval interface and a governance rule execution engine for the unified metadata caalog so that the unified metadata caalog can be accessed in the data weaving environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, in particular to an automatic metadata management and device for data weaving governance. BACKGROUND

[0002] In the digital transformation process of a data-driven enterprise, as the core carrier for centralized processing of multi-source and multi-type data, the structured metadata (including descriptive metadata describing data attributes and management metadata recording data ownership) generated in the data factory continues to expand with the expansion of business scenarios. At the same time, the data weaving architecture has become an important technical direction for enterprises to achieve global data governance due to its characteristics of cross-data source collaboration and flexible adaptation to multiple business scenarios. Under this background, how to realize efficient management of metadata and ensure its precise governance capability in the data weaving environment has become a core requirement to ensure data availability and support accurate business decision-making.

[0003] In the prior art, the metadata is first classified according to the physical storage path in the data factory, then the general governance rules are matched and bound with the classified metadata through a configuration table, and finally the metadata text file is output to the data weaving environment through a preset file transfer protocol to support access.

[0004] In this existing solution, on the one hand, the metadata classification based on the physical storage path cannot capture the business semantic association between metadata, resulting in that the governance rule binding can only meet the general scenario requirements and is difficult to adapt to the precise governance requirements of multiple business domains in the data weaving environment; on the other hand, when the data set in the data factory is changed, such as addition, modification or deletion, the state information and flow trail of the metadata cannot be automatically synchronized to the governance rule system, and intervention update is required, which not only causes deviation between the metadata retrieval result and the actual data state in the data weaving environment, but also causes the governance rule execution to lag behind the data change, affecting the timeliness and reliability of data weaving governance. SUMMARY

[0005] To solve the above technical problems, the present application provides an automatic metadata management and device for data weaving governance to at least alleviate the above technical problems.

[0006] The technical scheme provided by the embodiments of the present application is as follows: An automatic metadata management and device for data weaving governance, comprising: A semantic mounting unit configured to mount semantic labels to a structured metadata set of a data factory based on a semantic model of a data weaving architecture to generate enhanced semantic metadata and complete data lineage records. A rule binding unit is configured to bind governance rules to the enhanced semantic metadata and complete data lineage records based on data weaving to generate metadata governance lists bound with governance rules; A change updating unit is configured to update dataset state fields and corresponding flow paths of complete data lineage records to the metadata governance lists based on a change data capture mechanism when a dataset change in a data factory is monitored to generate a unified metadata catalog; An interface configuration unit is configured to configure a metadata retrieval interface and a governance rule execution engine for the unified metadata catalog to make the unified metadata catalog accessible in a data weaving environment.

[0007] The technical solution in the application has the following technical benefits: 1. To solve the problem in the background art that "metadata classification based on physical storage paths cannot capture business semantic association, resulting in difficulty in adapting governance rules to multi-business domain collaborative governance in data weaving": The semantic mounting unit mounts semantic tags to a structured metadata set of a data factory based on a semantic model of a data weaving architecture to generate enhanced semantic metadata and complete data lineage records. Compared with the traditional hierarchical classification method based on physical storage paths, the unit gives metadata business semantic attributes (such as business domain association and data ownership association) through the semantic model, so that the metadata can carry clear business context, providing a semantic basis for precise binding of subsequent governance rules. At the same time, the complete data lineage records can trace the flow track of metadata in the data factory, further supporting metadata association analysis among multiple business domains, so that the governance rules can better adapt to the governance needs of multi-business domain collaboration in the data weaving environment, and improve the adaptability of governance rules to business scenarios.

[0008] 2. To solve the problem in the background art that "general governance rules are difficult to meet the requirements of precise governance in data weaving through configuration binding": The rule binding unit binds governance rules to the enhanced semantic metadata and complete data lineage records based on data weaving federation governance rules to generate metadata governance lists bound with governance rules. Compared with the traditional method of configuring general governance rules, the federation governance rules can develop differentiated rules for the governance needs of different business domains under the data weaving architecture, and the binding process is based on metadata and lineage records carrying semantic information, avoiding the insufficient adaptation of general rules to special business scenarios, so that the governance rules can better meet the precise governance needs of the data weaving environment and reduce the governance deviation caused by the generality of rules.

[0009] 3. For the problem of "metadata state and flow track need to be updated when data set changes, resulting in retrieval deviation and governance lag" in the background art: The change updating unit monitors the data factory data set change based on the change data capture mechanism, synchronously updates the data set state field and the flow path of the corresponding complete data lineage record to the metadata governance list, and generates a unified metadata catalog. Compared with the traditional intervention update method, the change data capture mechanism can realize automatic monitoring of data set changes and metadata synchronization, and can update the changed state information (such as the state field of adding, modifying, and deleting corresponding) and the flow path to the governance list without intervention, which on the one hand reduces the deviation between the metadata retrieval result and the actual data state, and on the other hand avoids the governance rule execution lagging behind the data change, and improves the timeliness and reliability of data weaving governance.

[0010] 4. For the problem of "outputting metadata text files through file transfer protocol, which is difficult to adapt to data weaving environment access requirements" in the background art: The interface configuration unit configures the metadata retrieval interface and the governance rule execution engine for the unified metadata catalog, so that the unified metadata catalog can be accessed in the data weaving environment. Compared with the traditional preset file transfer protocol output text file method, the metadata retrieval interface can support precise retrieval of multiple modules (such as data pipeline scheduling module and data product access control module) in the data weaving environment, and the governance rule execution engine can directly drive the rules in the metadata governance list to execute, avoiding the problems of "low retrieval efficiency and difficult direct call of governance rules" in the text file access method, and further ensuring the continuity of metadata management and governance in the data weaving environment. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The structure of an embodiment of the application is an automatic metadata management and a device for data weaving governance.

[0012] Figure 2 The flowchart of an embodiment of the application is an automatic metadata management and a method for data weaving governance. DETAILED DESCRIPTION

[0013] As Figure 1As shown, an apparatus for automated metadata management and data weaving governance includes: a semantic mounting unit: for performing semantic tag mounting on a structured metadata set of a data factory based on a semantic model of a data weaving architecture to generate enhanced semantic metadata and complete data lineage records; a rule binding unit: for performing governance rule binding on the enhanced semantic metadata and complete data lineage records based on a data weaving federation governance rule to generate a metadata governance list bound with the governance rule; a change updating unit: for synchronously updating a data set state field and a corresponding flow path of the complete data lineage records to the metadata governance list based on a change data capture mechanism when a data set change of the data factory is monitored to generate a unified metadata catalog; and an interface configuration unit: for configuring a metadata retrieval interface and a governance rule execution engine for the unified metadata catalog to make the unified metadata catalog accessible in a data weaving environment.

[0014] Optionally, the semantic mounting unit is specifically configured to perform type-based semantic tag mounting on the structured metadata set of the data factory based on the semantic model of the data weaving architecture and determine a flow path of the structured metadata set in the data factory to determine the enhanced semantic metadata and the complete data lineage records based on the semantic tags and the flow path.

[0015] Specifically, the semantic mounting unit internally includes a metadata type identification module, a type-based tag mounting module, a flow path determination module, and a result integration generation module, which implement the specific process of "performing type-based semantic tag mounting on the structured metadata set of the data factory based on the semantic model of the data weaving architecture and determining a flow path of the structured metadata set in the data factory to determine the enhanced semantic metadata and the complete data lineage records based on the semantic tags and the flow path" as follows: Preferably, in the specific technical implementation of the metadata type identification module, first, the structured metadata set of the data factory is taken as the processing object, the attribute fields of each piece of metadata in the set (the attribute fields include three sub-fields of "data content attributes", "data management attributes", and "data correlation attributes") are parsed, and type identification is completed through field matching. Specifically, if a piece of metadata contains data content attribute sub-fields such as "data name", "data format", and "business meaning", it is determined to be "descriptive metadata"; if it contains data management attribute sub-fields such as "data creator", "department affiliation", and "access frequency", it is determined to be "management metadata"; and if it contains data correlation attribute sub-fields such as "field mapping relationship", "inter-table dependency rules", and "data aggregation logic", it is determined to be "structural metadata". Through this field matching method, the original structured metadata set is split into three intermediate results of "descriptive metadata subset", "management metadata subset", and "structural metadata subset", which will be the processing object of the type-specific label mounting module.

[0016] Preferably, in a scenario, when the type-specific label mounting module is implemented, first, the semantic model of the data weaving architecture is loaded (the model includes three core components of "business domain hierarchical vocabulary table", "data ownership association graph", and "entity association verification rules": the business domain hierarchical vocabulary table records the terminology system of each business domain, such as the "retail domain" containing sub-domain terms such as "goods management", "order fulfillment", and "user operation"; the data ownership association graph records the correspondence between departments, personnel, and data ownership, such as "e-commerce operation department-Li Si" corresponding to the ownership of "user order data"; and the entity association verification rules define the legal association types between different data entities, such as the "goods table" and the "inventory table" can be associated through "goods ID"), and then the "descriptive metadata subset", "management metadata subset", and "structural metadata subset" output by the metadata type identification module are taken as the processing object, and type-specific semantic label mounting is performed.

[0017] 1. The business domain semantic label mounting of descriptive metadata, with "descriptive metadata subset" as the processing object, extracts the "business meaning" field core term of each piece of descriptive metadata (such as the "business meaning" of a piece of descriptive metadata is "user's order placement behavior record in APP", and the core term is "user's order placement behavior"); convert the core term into a text vector (use Word2Vec algorithm, train vector model based on data weaving semantic model business domain corpus, vector dimension is a preset fixed value, such as 512 dimensions, each element in the vector represents the weight of the core term in a certain business semantic dimension); then calculate the cosine similarity of the text vector and each sub-domain term vector in the "business domain hierarchical glossary" (the similarity value ranges from 0 to 1), select the sub-domain term with the highest similarity and not less than a preset threshold (such as 0.7) as the "business domain semantic label", and mount it to the corresponding descriptive metadata to obtain "descriptive metadata mounted with business domain semantic label".

[0018] 2. The data ownership label mounting of management metadata, with "management metadata subset" as the processing object, extracts the "department" field information of each piece of management metadata (such as "e-commerce operation department"); match the field information with the "department node" in the "data ownership association graph", find the "ownership subject node" associated with the department node (such as "e-commerce operation department" is associated with "e-commerce operation department-Li Si" and "e-commerce operation department-data group" two ownership subject nodes); further filter the unique ownership subject according to the "data creator" field of the metadata (such as "data creator" is "Li Si", then filter out "e-commerce operation department-Li Si"), take the ownership subject as "data ownership label", and mount it to the corresponding management metadata to obtain "management metadata mounted with data ownership label".

[0019] 3. The data association relationship label mounting of structural metadata, with "structural metadata subset" as the processing object, extracts the "field mapping relationship" "inter-table dependency rule" field information of each piece of structural metadata (such as "field mapping relationship: order table. user ID → user table. user ID; inter-table dependency rule: order table depends on user table"); based on "entity association verification rules", verify the legality of the field information (such as verifying whether "order table. user ID → user table. user ID" conforms to the legal association type of "order entity-user entity"); if the verification is passed, take the legal association type (such as "user ID association" "order-user table dependency") as "data association relationship label", and mount it to the corresponding structural metadata to obtain "structural metadata mounted with data association relationship label".

[0020] The above three types of metadata subsets mounted with labels will be one of the inputs of the flow path determination module, which will be used for subsequent path verification.

[0021] Preferably, the specific implementation process of the flow path determination module is as follows: first, the processing area division of the data factory is determined (including the raw area, bronze area, silver area, gold area, and consumption area, wherein the raw area stores raw data and data that have not been processed, the bronze area stores metadata after preliminary cleaning, the silver area stores metadata after standardization processing, the gold area stores metadata after aggregation analysis, and the consumption area stores metadata for direct business calling). Based on the "descriptive metadata subset, management metadata subset, and structural metadata subset" and the "knowledge graph entity relationship in the semantic model of the data weaving architecture", the first step is to retrieve the "metadata entity nodes" of the three types of metadata subsets in the corresponding processing areas based on the knowledge graph entity relationship (which records the metadata entities of each processing area and the flow transfer association between entities, such as the "user raw behavior metadata" entity in the raw area can be transferred to the "user cleaned behavior metadata" entity in the bronze area). For example, the descriptive metadata subset corresponds to the "raw area-user behavior description metadata node" and the "bronze area-cleaned user behavior description metadata node", the management metadata subset corresponds to the "raw area-user behavior management metadata node" and the "bronze area-cleaned user behavior management metadata node", and the structural metadata subset corresponds to the "raw area-user behavior structure metadata node" and the "bronze area-cleaned user behavior structure metadata node". The entity nodes of the three types of metadata subsets are concatenated in chronological order to obtain an "initial flow transfer node sequence" (such as [raw area-descriptive meta node A, raw area-management meta node B, raw area-structural meta node C, bronze area-descriptive meta node A1, bronze area-management meta node B1, bronze area-structural meta node C1, silver area-descriptive meta node A2...]). The second step is to take the "initial flow transfer node sequence" and the "descriptive metadata with mounted business domain semantic labels" as the processing objects, and check the consistency of the descriptive metadata labels of adjacent areas in the initial flow transfer node sequence according to the "data domain association rule" in the data weaving semantic model (which stipulates that the metadata of adjacent processing areas should satisfy the consistency of business domain semantic labels, i.e., the matching degree of the business domain semantic label of the metadata of the previous area with the preset business domain label of the next area should not be lower than a preset threshold, such as 0.6). For example, the matching degree of the "business domain semantic label" (such as "retail domain-user operation") of the bronze area-descriptive meta node A1 with the preset business domain label (such as "retail domain-user operation") of the silver area is checked. If the matching degree is lower than the threshold, it is determined that there is a "data domain attribution conflict". At this time, according to the correction strategy in the data domain association rule (such as supplementing the "business domain conversion specification" field or adjusting the flow transfer direction to the node with a higher matching degree in the silver area, such as the "retail domain-user operation" node), the initial flow transfer node sequence is corrected to obtain a "corrected flow transfer node sequence".Thirdly, the nodes of the three types of metadata subsets in the modified flow-through node sequence are merged according to the processing area (for example, the description node A1, the management node B1 and the structure node C1 in the same bronze area are merged into a "bronze area-comprehensive metadata node"), to form a "complete flow-through path of the structured metadata set" (for example, [original area-comprehensive metadata node, bronze area-comprehensive metadata node, silver area-comprehensive metadata node, gold area-comprehensive metadata node, consumption area-comprehensive metadata node]).

[0022] Preferably, in the specific technical implementation of the result integration generation module, the description metadata with the business domain semantic label, the management metadata with the data ownership label, the structure metadata with the data association label and the complete flow-through path of the structured metadata set are taken as the processing objects. First, the three types of metadata subsets with the labels are aggregated through the "metadata unique identifier" (for example, the "metadata ID" of each piece of metadata), to form "enhanced semantic metadata" (the data contains the attributes and corresponding semantic labels of the three types of metadata, such as "metadata ID: MD001; data name: user order record; business domain semantic label: retail domain-user operation; data ownership label: e-commerce operation department-Li Si; data association label: user ID association").

[0023] Subsequently, the "enhanced semantic metadata" is associated with the "complete flow-through path" through the "processing area unique identifier" (for example, the "area ID" of each area), so that each "comprehensive metadata node" in the complete flow-through path is bound to the enhanced semantic metadata segment of the corresponding area (for example, the bronze area-comprehensive metadata node is bound to the enhanced semantic metadata segment of the bronze area), to form a "flow-through path record containing semantic labels"; finally, the "path node timestamp" (for example, the entering time and leaving time of each node) is supplemented in the record, to obtain a "complete data lineage record" (the record contains the flow-through path, the semantic labels of each node and the timestamp, such as "original area-comprehensive metadata node (time: T1; label: retail domain-user operation...) → bronze area-comprehensive metadata node (time: T2; label: retail domain-user operation...) →...").

[0024] The implementation process is different from the traditional metadata processing method based on "physical storage path" in three aspects: first, through "metadata type splitting + type-specific semantic label mounting", relying on the semantic model of data weaving to give metadata business context, solving the problem of lack of semantic association and difficulty in adapting to multi-business domain collaborative governance of traditional metadata; second, combined with "knowledge graph entity relationship + data domain association rule" to determine the flow path, ensuring that the path not only reflects the physical flow, but also meets the semantic consistency requirements of data weaving, avoiding the governance deviation caused by semantic conflict of traditional path; third, through multi-dimensional identifiers (metadata ID, region ID), the deep association of labels and paths is realized, so that the complete data lineage record generated has semantic explainability and path coherence, which is more suitable for the scene needs of "semantic driven governance" in the data weaving environment. Based on the above module division and step description, a person skilled in the art can realize the functions of each module through conventional programming tools (such as Python), for example, metadata type identification based on Pandas library, text vector conversion based on Gensim library, and knowledge graph entity relationship retrieval based on Neo4j.

[0025] Optionally, the structured metadata set includes descriptive metadata, management metadata, and structural metadata divided by type, and the semantic mounting unit mounts a business domain semantic label for the descriptive metadata, a data ownership label for the management metadata, and a data association relationship label for the structural metadata when performing type-specific semantic label mounting on the structured metadata set.

[0026] Optionally, the semantic model of the data weaving architecture includes data domain association rules and knowledge graph entity relationships, and the semantic mounting unit determines the flow path of the structured metadata set in the data factory based on the knowledge graph entity relationships to preliminarily locate the flow nodes of the structured metadata set in each processing area of the data factory, and then checks the consistency of the data domain attribution between the flow nodes according to the data domain association rules. For the flow nodes with data domain attribution conflict, the flow direction is corrected according to the data domain association rules to trace the flow path of the structured metadata set.

[0027] Preferably, the semantic mounting unit implements the specific process of "mounting the structured metadata set with typed semantic labels" as follows: the core completes the three steps of "metadata type identification, typed label generation, and label binding integration" to ensure that the label is deeply adapted to the metadata type and data weaving scene. First, perform structured metadata type identification to obtain three subsets of metadata. Taking the "structured metadata set of the data factory" as the processing object, parse the "core attribute field" of each metadata: if the metadata contains the "business meaning", "data format", and "data source description" fields (used to describe the data content attribute), it is determined as "descriptive metadata"; if it contains the "department to which the data belongs", "creator", and "access permission level" fields (used to describe the data management attribute), it is determined as "management metadata"; if it contains the "field mapping relationship", "inter-table dependency rule", and "data aggregation logic" fields (used to describe the data correlation attribute), it is determined as "structural metadata". After type division through field matching, generate "descriptive metadata subset", "management metadata subset", and "structural metadata subset", each of which carries a "metadata unique ID" (used for subsequent association). Second, mount the business domain semantic label for descriptive metadata. Taking the "descriptive metadata subset" and the "semantic model of the data weaving architecture (including the business domain hierarchical vocabulary)" as the processing object, extract the core term of the "business meaning" field of each descriptive metadata (such as "user APP order record" extracting "user APP order"); convert the core term into a semantic vector (using the Word2Vec algorithm, trained based on the business domain corpus, with a vector dimension of a reasonable value adapted to the business semantics, specific examples being 512 dimensions, and vector elements representing the weight of the term in the "business domain-subdomain-scene" dimension); calculate the cosine similarity of the vector and the term vector in the business domain hierarchical vocabulary (such as "retail domain" containing "user operation" and "order fulfillment" subdomains); select the subdomain term with a similarity not lower than a preset threshold (the general angle is to "ensure the business domain matching degree", specific examples being 0.6) as the "business domain semantic label" (such as "retail domain-order fulfillment-APP order"), and mount it to the "semantic label" field of the descriptive metadata to obtain "descriptive metadata with mounted business domain semantic labels". Then, mount the data ownership label for management metadata.The semantic model of the data weaving architecture (including knowledge graph entity relationship) is taken as the processing object, the knowledge graph entity relationship includes a department node, a personnel node, a data type node, and an ownership association edge (for example, the "E-commerce operation department node" is associated with the "user order data node" through the "ownership" edge); the "data belonging department" and "creator" fields of the management metadata are extracted, the "department node" of the knowledge graph is matched (for example, the "E-commerce operation department" is matched with the "E-commerce operation department node"), and then the "personnel node" associated with the department node is filtered through the "creator" field (for example, the "E-commerce operation department-Zhang San node"); the "department-personnel-data type" combination (for example, the "E-commerce operation department-Zhang San-user order data") is taken as the "data ownership label" and is mounted to the "ownership identifier" field of the management metadata, and the management metadata with the mounted data ownership label is obtained. Then, the data association relationship label is mounted for the structural metadata. The semantic model of the data weaving architecture (including entity association rule) is taken as the processing object, the entity association rule defines a legal association type (for example, the "order table.user ID" and the "user table.user ID" can be associated); the "field mapping relationship" and "table interdependence rule" fields of the structural metadata are extracted, and whether they conform to the entity association rule is checked (for example, the "goods table.goods ID→stock table.goods ID" conforms to the "goods-stock entity association" rule); the legal association type (for example, the "user ID field association" and the "order table depends on the user table") is combined into the "data association relationship label" and is mounted to the "association rule" field of the structural metadata, and the structural metadata with the mounted data association relationship label is obtained. Finally, the three types of metadata with mounted labels are summarized, are associated through the "metadata unique ID", and a set of enhanced semantic metadata with completed type semantic labels is generated. The set will be taken as the core input of the subsequent "determining the flow path" step, and the business domain, ownership, and association relationship label carried by the set provide a basis for semantic verification of the flow path, which is different from the traditional "general label mounting" mode, and realizes accurate adaptation of the metadata label to the type and business scenario.

[0028] Preferably, the semantic mounting unit determines the specific technical implementation of the flow path based on the "semantic model of data weaving architecture (data domain association rule, knowledge graph entity relationship)". The core forms a closed loop through "initial positioning → consistency verification → conflict correction → path generation", ensuring that the path has both physical flow and semantic consistency. First, the knowledge graph entity relationship is used to preliminarily locate the flow node. The "enhanced semantic metadata set mounted with type-based semantic tags" and the "knowledge graph entity relationship" are used as the processing object. The knowledge graph entity relationship includes "processing area nodes" (original area, bronze area, silver area, gold area, and consumer area nodes, including the "adapted business domain" attribute) and "metadata entity nodes" (such as "retail domain user metadata nodes", associated with "metadata unique ID"). The enhanced semantic metadata is associated with the "metadata entity nodes" through the "metadata unique ID", and the "flow association edge" of the "metadata entity nodes" (such as the "retail domain user metadata nodes" and the "original area node" "bronze area node" "flowable" edge) is used to match the corresponding "processing area nodes" of each metadata. According to the time sequence rule of the data factory "original area → bronze area → silver area → gold area → consumer area", the processing area nodes are sorted to form an "initial flow node sequence of structured metadata set" (such as [original area-retail domain user node, bronze area-retail domain user node, silver area-retail domain user node]). Second, the data domain association rule is used to verify the consistency of the data domain attribution between the flow nodes. The "initial flow node sequence", "data domain association rule", and "descriptive metadata mounted with business domain semantic tags" are used as the processing object. The data domain association rule specifies the "business domain tag matching degree threshold value" of the adjacent processing area nodes (the general angle is "to ensure semantic consistency", and the specific example is 0.6) and the "matching degree calculation method" (based on the cosine similarity of the business domain semantic tags). The adjacent node pairs in the initial sequence (such as [original area node, bronze area node]) are traversed, and the "business domain semantic tags" associated with each node are extracted (obtained from the descriptive metadata, such as the original area node tag "retail domain-user operation" and the bronze area node tag "retail domain-user operation"). The tag matching degree is calculated, and if the matching degree is greater than or equal to the threshold value, it is marked as a "verification passed node pair". If the matching degree is less than the threshold value (such as the bronze area node tag "financial domain-credit" with a matching degree of 0.3), it is marked as a "data domain attribution conflict node pair", and the conflict reason ("bronze area node business domain does not match the original area") is recorded. The "flow node verification result table" is generated. Subsequently, the data domain association rule is used to correct the data domain attribution conflict flow node.With "flow conversion node verification result table" and "data domain association rule (including conflict correction strategy)" as the processing object, the conflict correction strategy is divided into two categories: one is "semantic conversion supplement", if the business domain label of the conflict node exists partial association (such as the original area "retail domain-user operation" and the bronze area "retail domain-order fulfillment"), supplement "business domain conversion description" (such as "user operation data is converted into order fulfillment data after order association"), embed "association attribute" of bronze area node, get "bronze area node with supplementary conversion description"; the second is "flow conversion node adjustment", if the conflict node has no association label (such as the original area "retail domain" and the bronze area "financial domain"), search "candidate metadata entity node" (such as "bronze area-retail domain special node") associated with the bronze area node in the knowledge graph, replace the original bronze area node, get "adjusted bronze area node"; put the corrected node back into "flow conversion node verification result table" to calculate the matching degree, until it passes the verification, generate "corrected flow conversion node sequence". Finally, generate the complete flow path of the structured metadata set. With "corrected flow conversion node sequence" and "enhanced semantic metadata set" as the processing object, supplement "flow time stamp" (system time when metadata enters the area) and "associated metadata ID" (bind enhanced semantic metadata) for each node; according to "time stamp in ascending order", concatenate the nodes, merge multiple nodes in the same processing area into "area comprehensive node" (such as "supplementary conversion description node" and "user metadata node" in the bronze area are merged into "bronze area-retail domain comprehensive node"); finally form "complete flow path containing semantic consistency" (such as "original area-retail domain comprehensive node (T0) → bronze area-retail domain comprehensive node (T1, including conversion description) → silver area-retail domain comprehensive node (T2) → gold area-retail domain comprehensive node (T3) → consumption area-retail domain comprehensive node (T4)"). The path is different from the traditional "path based only on physical storage". Through semantic verification and conflict correction, it ensures that the metadata meets the business domain requirements when flowing through each area, and provides a semantic path basis for subsequent generation of complete data lineage records.

[0029] In the above scheme, ① classification type semantic label mounting: unlike the traditional "general label without distinction", the core attributes (content, management, association) of descriptive, management, and structural metadata are designed with differentiated labels, combined with the business domain vocabulary table of data weaving semantic model, knowledge graph, and entity rules, to realize the deep adaptation of labels and metadata functions, and avoid the lack of governance precision caused by general labels; ② semantic flow path determination: break through the traditional "path based on physical storage directory positioning", associate metadata with processing areas through knowledge graph, and then verify semantic consistency with data domain association rule, conflict correction strategy ensures that the path has both physical flow and business semantic coherence, solves the problem of disconnection between traditional path and business scenario.

[0030] Optionally, the rule binding unit is specifically configured to: perform label quality and path integrity verification on the enhanced semantic metadata and the complete data lineage record respectively to obtain metadata and lineage records that pass the quality verification; and perform scene-based governance rule binding on the metadata and the lineage records that pass the quality verification based on the data weaving federation governance rule, to generate a metadata governance list bound with the governance rule.

[0031] In particular, the rule binding unit internally contains a quality verification module, a scene-based rule binding module, and a governance list generation module, which realize the specific process of "respectively performing label quality and path integrity verification on enhanced semantic metadata and complete data lineage records to obtain metadata and lineage records that pass the quality verification; based on the data weaving federation governance rules, performing scene-based governance rule binding on the metadata and lineage records that pass the quality verification to generate metadata governance lists bound by governance rules" as follows: The core pre-step of the rule binding unit is to ensure the quality of the input data to avoid governance rule binding deviation caused by low-quality metadata or lineage records. The quality verification module is divided into two sub-steps: "enhanced semantic metadata label quality verification" and "complete data lineage record path integrity verification", and the specific implementation is as follows: Preferably, the specific implementation process of the enhanced semantic metadata label quality verification in the quality verification module is as follows: First, taking "enhanced semantic metadata generated by the semantic mounting unit" as the processing object, it is divided into three sub-sets of "descriptive metadata mounted with business domain semantic labels", "management metadata mounted with data ownership labels", and "structural metadata mounted with data association relationship labels" according to the metadata type, and the "label verification dimension" of each type of metadata is determined (the descriptive metadata verifies the "business domain semantic label", the management metadata verifies the "data ownership label", and the structural metadata verifies the "data association relationship label") to obtain a "classified metadata label verification list". Second, perform label quality verification for each type of metadata: for "descriptive metadata mounted with business domain semantic labels", verify "label existence" (whether the business domain semantic label is missing), "label consistency" (the semantic matching degree of the label and the "business meaning" field of the metadata, calculate the cosine similarity, and the general angle is "not less than a reasonable threshold", and a specific example is 0.6), and "label association" (whether the label exists in the "business domain hierarchical vocabulary table" of the data weaving architecture semantic model), if all three are passed, it is marked as "descriptive metadata label verification passed"; for "management metadata mounted with data ownership labels", verify "label integrity" (whether the label contains "department-personnel-data type" three elements, such as "e-commerce operation department-Zhang San-user order data") and "label validity" (whether the ownership subject in the label exists in the "data ownership association graph" of the data weaving semantic model), both of which are passed to mark as "management metadata label verification passed"; for "structural metadata mounted with data association relationship labels", verify "label legality" (whether the association relationship in the label conforms to the "entity association verification rule" of the semantic model, such as "order table.commodity ID "Tag Quality Check" (whether the "Inventory Table. Commodity ID" is a legal association), "Tag Integrity" (whether it contains "Field Association + Table Dependency" two types of sub-labels), both pass, marked as "Structural Metadata Tag Check Passed". Finally, the check results of the three types of metadata are summarized, and the enhanced semantic metadata set with all tag check passed is generated. This result will be one of the core inputs of the scene rule binding module, ensuring that the subsequent rule binding is based on semantic compliant metadata.

[0032] Preferably, the specific technical implementation of the complete data lineage record path integrity check in the quality check module is as follows: first, taking the "complete data lineage record generated by the semantic mounting unit" as the processing object, extracting the core constituent elements of the record: "flow transfer node sequence" (such as [original area-comprehensive metadata node, bronze area-comprehensive metadata node, silver area-comprehensive metadata node]), "node conversion rule" (metadata conversion logic associated with each flow transfer node, such as "bronze area node-data cleaning logic"), "path timestamp" (entry / exit time of each node, such as original area T0, bronze area T1), to get "lineage record core element list". Secondly, the core elements are checked for integrity: check "flow transfer node integrity": check whether the node sequence covers all necessary processing areas of the data factory (original area, bronze area, silver area, gold area, consumption area), if a certain area node is missing (such as missing silver area node), mark it as "node missing"; check "node sequence rationality": check whether the node sequence meets the time sequence rule of the data factory "original area→bronze area→silver area→gold area→consumption area", if there is a reverse sequence (such as bronze area→original area), mark it as "sequence abnormal"; check "conversion rule existence": check whether each flow transfer node is associated with the corresponding "node conversion rule", if a certain node (such as bronze area node) has no conversion rule, mark it as "conversion rule missing"; check "timestamp continuity": check whether the timestamps of adjacent nodes meet the condition "previous node exit time≤next node entry time" (such as bronze area entry time T1≥original area exit time T0), if not, mark it as "timestamp fracture". Subsequently, the lineage records with check abnormalities are modified (such as missing silver area node, based on the "data domain association rule" of the semantic model to supplement the "silver area-comprehensive metadata node" and the corresponding conversion rule), and then rechecked until all elements meet the integrity requirements, generating "complete data lineage record with path integrity check passed". This result will be one of the inputs of the scene rule binding module together with "enhanced semantic metadata set with tag quality check passed", ensuring that the rule binding relies on compliant metadata and lineage path at the same time.

[0033] Preferably, the specific implementation process of the scene-based rule binding module is as follows, which is based on metadata type + lineage flow scene to realize accurate rule matching, abandoning the traditional general rule binding mode: First, taking the "data weaving federal governance rule" as the processing object, analyze its core components: one is "data domain access permission rule" (define different business domains, different ownership subjects of metadata access permission, such as "retail domain- e-commerce operation department personnel can read and write retail domain user metadata, other departments can only read"), two is "data product life cycle management rule" (define the whole life cycle control logic of metadata from generation to archiving, such as "the retention period of raw area metadata is 90 days, and the retention period of gold area metadata is permanent"), and clearly define the "adapted metadata type" of the two types of rules - the data domain access permission rule is adapted to "management metadata" (because management metadata records ownership information), and the data product life cycle management rule is adapted to "structural metadata" (because structural metadata records association relationship, which affects the life cycle flow), and get the "rule-metadata type adaptation table". Secondly, taking the "enhanced semantic metadata set passed the label quality check", "complete data lineage record passed the path integrity check" and "rule-metadata type adaptation table" as the processing object, execute the scene-based rule binding: management metadata and data domain access permission rule binding: extract the "data ownership label" in the "management metadata mounted with data ownership label" (such as "e-commerce operation department-Zhang San-user order data"), match the corresponding relationship of "ownership subject-business domain-permission" in the data domain access permission rule (such as "e-commerce operation department-retail domain-read and write permission"), embed the matched rule into the "permission constraint" field of the management metadata, and adjust the rule effective range combined with the "flow node" of the complete data lineage record (such as "only read and write permission in gold area and consumption area, only read permission in raw area-silver area"), get "management metadata bound with access permission rule"; structural metadata and data product life cycle management rule binding: extract the "data association relationship label" in the "structural metadata mounted with data association relationship label" (such as "order table. commodity ID The inventory table "commodity ID" is matched with the corresponding relationship of "association type-life cycle parameter" in the data product life cycle management rule (such as "order-inventory association metadata-raw area reservation 90 days, gold area permanent reservation"), the matched rule is embedded in the "life cycle constraint" field of the structural metadata, and the "node conversion rule" of the complete data lineage record is combined to supplement the life cycle trigger condition (such as "start life cycle timing when the bronze area node performs 'data cleaning' conversion"), to obtain "structural metadata bound with life cycle rule"; rule association of descriptive metadata: the "business domain semantic label" (such as "retail domain-user operation") of the descriptive metadata is taken as an "association index", and is associated to the access permission rule of the corresponding management metadata and the life cycle rule of the structural metadata (such as "retail domain-user operation" descriptive metadata is associated with "e-commerce operation department read-write permission" "90 days-permanent reservation" rule), to obtain "descriptive metadata associated with governance rule". Finally, the metadata after binding of the three types of rules is summarized to generate "metadata set after binding of rules in different scenes". In the result, each type of metadata is bound with the governance rule suitable for its type and flow transfer scene, solving the problem of insufficient adaptability of traditional general rules to special business scenes.

[0034] Preferably, the specific implementation of the governance list generation module is as follows: first, taking the "metadata set after scene rule binding" and the "complete data lineage record after path integrity verification" as the processing object, extract the "core identification information" of each type of metadata, including the "metadata unique ID" (such as "MD-Desc-001") and "business domain semantic label" of descriptive metadata, the "metadata unique ID" and "data ownership label" of management metadata, the "metadata unique ID" and "data correlation relationship label" of structural metadata, and the "flow path identification" (such as "Lineage-Path-001") of the complete data lineage record, to obtain the "metadata lineage association identification set". Second, taking the "metadata lineage association identification set" as the index, construct the "metadata governance list entry", each entry contains the following structured fields: metadata basic information: metadata unique ID, metadata type (descriptive / management / structural), associated lineage path identification; binding governance rules: rule type (data domain access permission / data product life cycle), rule content (such as "read-write permission + gold zone-consumption zone effective"), rule effective condition (such as "bronze zone data cleaning trigger"); quality check result: label quality check result (such as "label existence / consistency / association all pass"), path integrity check result (such as "node complete / reasonable order"). Subsequently, sort all "metadata governance list entries" by "metadata type" (descriptive metadata→management metadata→structural metadata), and add "list version number" (such as "V1.0"), "generation timestamp" (such as "2024-10-03T14:30:00") and "rule binding responsible person" (such as "data governance group-Li Si") and other management fields to obtain the "metadata governance list with binding governance rules". Finally, store the list and the "metadata set after scene rule binding" and the "complete data lineage record" through the "metadata unique ID" and "lineage path identification", to ensure that the "change update unit" can quickly locate the governance rules corresponding to the metadata through the list, and realize the linkage update of rules and metadata.

[0035] Different from the traditional "generalized governance rule binding" (binding the same rule to all metadata without distinguishing types and scenarios), the scheme is different in that: ①Quality pre-check: Through "label quality + path integrity" two-dimensional check, avoid the rule binding deviation caused by low-quality metadata / lineage record, solve the problem of "rebinding, light quality" of traditional rule binding; ②Precise binding by scene: Based on the matching rules of metadata type (administrative-> access permission, structural-> life cycle) and lineage flow scenario (such as gold zone-consumption zone effective permission), realize the three-dimensional adaptation of "metadata type-rule type-flow scenario", improve the fit degree of rules and business needs; ③Lineage correlation adjustment: The flow node and conversion rule of the lineage record are used as the basis for adjusting the effective range of the rule, so that the rule is not only bound to the metadata itself, but also adapted to its actual flow path in the data factory, avoiding the defects of traditional rules "static binding, out of scene".

[0036] Optionally, the data weaving federation governance rule includes a data domain access permission rule and a data product life cycle management rule, and the rule binding unit binds the data domain access permission rule to the administrative metadata in the structured metadata set and binds the data product life cycle management rule to the structural metadata in the structured metadata set when performing scene-based governance rule binding on the metadata and the lineage record that pass the quality check, to generate a metadata governance list that binds the governance rule.

[0037] Specifically, the rule binding unit performs the process of "binding the data domain access permission rule to the administrative metadata and binding the data product life cycle management rule to the structural metadata", relying on the "rule analysis module", "metadata-rule matching module" and "binding relationship solidification module", and the specific implementation is as follows: Preferably, the specific implementation process of the rule analysis module for structurally disassembling the data weaving federal governance rules is as follows: first, taking the "data weaving federal governance rules" as the processing object, splitting it into the "data domain access permission rule set" and the "data product life cycle management rule set" according to the rule type, and clearly defining the core governance targets of the two types of rules: the data domain access permission rule focuses on "who can access what data under what conditions", and the data product life cycle management rule focuses on "how data should be stored, transferred and destroyed in each stage". Secondly, the "data domain access permission rule set" is extracted: each rule is disassembled into "permission subject" (such as "retail domain-e-commerce operation department" and "financial domain-risk control team"), "target data domain" (such as "retail domain-user order" and "financial domain-credit approval"), "permission type" (read / write / modify / delete), "effective scene constraint" (such as "effective only in the gold zone and consumption zone" and "effective after being approved by the data owner"), forming a "data domain access permission rule element table" (each row corresponds to a rule, and the columns correspond to the above elements). Subsequently, the "data product life cycle management rule set" is extracted: each rule is disassembled into "association relationship type" (such as "order table-user table association" and "inventory table-goods table association"), "life cycle stage" (original zone storage stage / bronze zone cleaning stage / silver zone standardization stage / gold zone aggregation stage / consumption zone application stage), "stage control parameter" (storage time, quality verification frequency, backup strategy), "stage conversion trigger condition" (such as "when cleaning is completed and the quality verification pass rate is greater than or equal to the preset threshold, enter the next stage"), forming a "data product life cycle management rule element table". Finally, the two element tables are associated with the semantic model of the data weaving architecture (such as "target data domain" associated with the "business domain hierarchical vocabulary table" of the semantic model, and "association relationship type" associated with the "entity association rule" of the semantic model), to ensure that the rule elements meet the semantic specifications, and obtain a "set of governance rule elements that meet the semantics". This set provides a standardized rule benchmark for subsequent metadata-rule matching.

[0038] Preferably, the specific technical implementation of the management metadata in the metadata-rule matching module and the data domain access permission rule matching binding is as follows: first, taking "quality checked management metadata" as the processing object, extracting the core elements of "data ownership label" - "belonging business domain" (such as "retail domain"), "ownership subject" (such as "e-commerce operation department-Zhang San"), and "management data type" (such as "user order data") to form "management metadata feature identifier" (such as "retail domain-e-commerce operation department-user order data"). Secondly, taking "management metadata feature identifier" as the search key, perform matching in the "data domain access permission rule element table" of the "semantic compliant governance rule element set": first dimension matches the business domain consistency of "belonging business domain" and rule "target data domain" (such as "retail domain" matching "retail domain-user order"); second dimension matches the ownership relationship of "ownership subject" and rule "permission subject" (such as "e-commerce operation department-Zhang San" matching "retail domain-e-commerce operation department"); third dimension matches the data type compatibility of "management data type" and rule "target data domain" (such as "user order data" matching the data type of "retail domain-user order"). Through three-dimensional matching, one or more data domain access permission rules with the highest matching degree are screened out (the general angle is "matching dimension ≥ 2", and the specific example is three-dimensional full matching), and a "candidate access permission rule set" is obtained. Then, combine the "path integrity checked complete data lineage record" to optimize the candidate rules: extract the "flow node sequence" of the lineage record (such as [raw area, bronze area, gold area, consumption area]), and adjust the "effective scene constraint" of the rule according to the "data sensitivity level" of the node (raw area→consumption area, increasing sensitivity), such as adding "second approval" constraint for high sensitivity gold area / consumption area; for low sensitivity raw area / bronze area, the rule is relaxed to "read only"), and obtain "access permission rule adapted to lineage scene". Finally, embed "access permission rule adapted to lineage scene" into the "permission governance field" of the management metadata (field structure is "rule ID+permission type+effective node+constraint condition"), and generate "management metadata bound with data domain access permission rule".

[0039] Preferably, in a scene, the specific implementation of the structural metadata in the metadata-rule matching module and the data product lifecycle management rule matching binding is as follows: first, taking "quality checked structural metadata" as the processing object, extracting the core elements of "data correlation relationship label" - "main table name" (such as "order table"), "correlation table name" (such as "user table"), and "correlation field pair" (such as "order table.user_id User Table.id”), “strength of association” (strong / weak association, determined based on field matching degree), combined to form “structural metadata association feature” (e.g. “Order Table-User Table-strong association (user_id id)”). Secondly, taking “structural metadata association feature” as the search key, perform matching in the “Data Product Lifecycle Management Rule Element Table” of “Semantic Compliance Governance Rule Element Set”: first dimension matches the consistency of “main table-association table combination” and rule “association relationship type” (e.g. “Order Table-User Table” matches “Order Table-User Table Association”); second dimension matches the adaptability of “strength of association” and rule “stage control parameter” (e.g. strong association corresponds to the parameter of “storage time length extended by 30%”, weak association corresponds to “standard storage time length”). Through two-dimensional matching, filter out a unique matching data product lifecycle management rule, and obtain “candidate lifecycle rule”. Then, combine “complete data lineage record passed path integrity verification” to optimize the candidate rule: extract “conversion rule of flow transfer node” (e.g. “data cleaning logic” in bronze area, “field standardization rule” in silver area) in the lineage record, bind “stage conversion trigger condition” of the rule with the conversion rule (e.g. “when the bronze area node completes the conversion of ‘data cleaning logic and null rate ≤ 5%’, trigger the lifecycle stage conversion from bronze area to silver area”), and obtain “lifecycle rule adapted to lineage conversion scene”. Finally, embed “lifecycle rule adapted to lineage conversion scene” into the “lifecycle governance field” of structural metadata (field structure is “rule ID+stage sequence+parameters of each stage+conversion trigger condition”), and generate “structural metadata bound with data product lifecycle management rule”.

[0040] Preferably, the binding relationship solidification module generates the specific implementation process of the metadata governance list as follows: first, taking "binding data domain access permission rule management metadata" and "binding data product life cycle management rule structural metadata" as the processing object, extracting the "metadata unique identifier" (such as "MD-Mgmt-001" and "MD-Struct-001"), "binding rule ID" (such as "Rule-Access-001" and "Rule-Lifecycle-001"), and "associated lineage record ID" (such as "Lineage-001"), and forming a "metadata-rule association index table". Secondly, based on the "metadata-rule association index table", the "metadata governance list basic entry" is constructed: each entry contains "metadata type" (management / structure), "metadata unique identifier", "binding rule details" (rule type + core element + effective constraint), "associated lineage node" (rule effective flow node), and "matching verification result" (matching dimension + matching degree). Subsequently, "descriptive metadata" is introduced as an associated link: the "business domain semantic label" (such as "retail domain-user order") of "quality verification passed descriptive metadata" is associated with the "belonging business domain" of management metadata and the "main table business domain" of structural metadata, so that each list entry realizes the linkage of "business semantics-management permission-life cycle" through descriptive metadata, forming a "three-dimensional associated metadata governance list entry". Finally, all "three-dimensional associated metadata governance list entries" are sorted by "business domain" and "metadata type" in two levels, and "list generation timestamp", "rule version number", "binding person in charge" and other management fields are added, to generate a "metadata governance list bound with governance rules". In this list, the binding relationship between management metadata and access permission rules and between structural metadata and life cycle rules is solidified through three associations of "metadata unique identifier", "rule ID", and "lineage record ID", ensuring that the associated rules can be accurately located during subsequent changes and updates.Different from the traditional "static binding of rules and metadata" (no distinction between metadata types, fixed rule content), the scheme is different in that: ①Type accurate binding: based on the essential difference between management metadata (carrying ownership information) and structural metadata (carrying association relationship), respectively matching access permission rules and life cycle rules, solving the problem of insufficient adaptability caused by traditional general rules "one size fits all"; ②Lineage scene dynamic adaptation: the flow node and conversion rule of lineage record are used as the adjustment basis of rule effectiveness constraints (such as strengthening permission control in gold zone, triggering life cycle conversion after cleaning is completed), so that the rule is upgraded from "static text" to "dynamic constraint adapting to data flow scene"; ③Three-dimensional association mechanism: through the business domain semantic label of descriptive metadata, the linkage binding of "business semantics-management permissions-life cycle" is realized, ensuring that the rule not only restricts the metadata itself, but also deeply couples with the business scene.

[0041] Optionally, the change updating unit is specifically configured to: based on a change data capture mechanism, monitor a change type of a data set in the data factory in real time; when the change of the data set is monitored, synchronously collect a change reason log, and update a data set state field according to the change type, and update a flow path of a complete data lineage record, and synchronously update the updated field and the path to a metadata governance list, to generate a unified metadata catalog.

[0042] Optionally, the change type includes adding, modifying, and deleting; when the change updating unit monitors the change of the data set, the change reason log including a change initiator and a change operation description is synchronously collected; when the data set state field is updated according to the change type, the corresponding "to be checked" is added, the corresponding "updated" is modified, and the corresponding "archived" is deleted; when the flow path of the complete data lineage record is updated, the initial flow node of the data set is added, the corresponding conversion node of the data set is modified, and the terminal node of the data set is marked; and the updated data set state field and the flow path are synchronously updated to the metadata governance list, to generate the unified metadata catalog.

[0043] Specifically, the change updating unit internally contains a change monitoring module, a change information collecting module, a state and path updating module, and a synchronization and catalog generating module, which realize the specific process of "monitoring dataset changes based on change data capture mechanism, collecting change reason logs, updating state fields and lineage paths, and synchronously generating unified metadata catalog" as follows: Preferably, the specific implementation process of the change monitoring module in the change updating unit is as follows: First, taking "dataset operation logs of each processing area (raw area, bronze area, silver area, gold area, and consumption area) of the data factory" as the processing object, a change data capture (CDC) mechanism is deployed - a "log parsing CDC" (different from an intrusive trigger CDC, avoiding affecting the read-write performance of the data factory), which parses the dataset operation logs of each area (such as "data write logs" of the raw area, "field update logs" of the bronze area, and "data deletion logs" of the consumption area) in real time, and extracts the "operation type identifier" (such as "INSERT" for addition, "UPDATE" for modification, and "DELETE" for deletion) in the logs. Secondly, for different operation type identifiers, define change type judgment rules: if the operation type identifier is "INSERT" and there is a "new data write path" (such as " / raw / retail / 20241003 / " of the raw area) in the log, it is determined as "addition" change type; if the operation type identifier is "UPDATE" and there is a "field modification record" (such as "user table.age field is updated from 25 to 26") in the log, it is determined as "modification" change type; if the operation type identifier is "DELETE" and there is a "data deletion marker" (such as "logical deletion marker = 1" or "physical deletion path") in the log, it is determined as "deletion" change type. Subsequently, a "change event identifier" (format: "change type-time stamp-transaction ID", such as "addition-20241003150000-TX001") is generated for each determined change type, and the "metadata unique ID" (such as "MD-001") in the log is associated to obtain a "structured change event set" (each event contains change type, change event identifier, metadata unique ID, and operation log segment). This set provides accurate change positioning basis for subsequent change information collection and updating.

[0044] Preferably, the specific technical implementation of the change information collection module in the change update unit is as follows: first, taking the "change event identifier" in the "structured change event set" as the retrieval key, the "operation audit log" (recording all data operation initiators, operation content, and timestamp) of the data factory is associated - association is achieved through "transaction ID" (TX001 in the change event identifier), ensuring that the change event and the operation audit log correspond one by one. Secondly, extract the "change reason core elements" from the associated operation audit log: "change initiator": such as "data engineer - Wang Wu" and "ETL scheduling service" (distinguishing between operations and system automatic operations); "change operation description": such as "add 20241003 retail order data", "modify user table mobile phone field format", and "delete expired 2023 financial credit data"; "change timestamp": accurate to milliseconds, consistent with the timestamp in the change event identifier, ensuring time sequence accuracy. Then, combine these elements with the "change event identifier" and "metadata unique ID" to generate the "structured change reason log" (fields include: log ID, change event identifier, metadata unique ID, initiator, operation description, and change timestamp), and establish permanent association with the change event through "log ID" to obtain the "structured change reason log set". This set avoids the problem of "reason record missing" in traditional change updates, providing complete basis for subsequent tracing of change background.

[0045] Preferably, the specific implementation of the data set state field update in the state and path update module is as follows: first, taking the "structured change event set" and "structured change reason log set" as the processing object, the corresponding "to-be-updated metadata" (i.e. the metadata that has changed, such as newly added retail order metadata, modified user metadata, and deleted credit metadata) is associated through the "metadata unique ID".

[0046] Secondly, update the "status field" of "metadata to be updated" according to the change type (the status field is located in the "management attribute" subfield of the metadata, and the initial default value is "normal"): if the change type is "add": update the status field to "to be checked", and fill in the "log ID" of the corresponding structured change reason log in the "associated log ID" attribute of the status field, to ensure that the subsequent quality checking process can be traced back to the change initiation background; if the change type is "modify": update the status field to "updated", and fill in the change operation description (such as "modify the format of the mobile phone field") in the "status note" attribute, and associate the change reason log ID; if the change type is "delete": update the status field to "archived", and fill in the change timestamp in the "archiving time" attribute, and associate the change reason log ID to explain the basis for deletion (such as "expired data cleaning"). Finally, select all the metadata whose status field is updated to generate a "metadata set with updated status field". This set ensures that the metadata status is synchronized with the actual change behavior in real time, avoiding the "status lag" problem caused by traditional updates.

[0047] Preferably, the state and path update module records the complete data lineage record flow path update process as follows: First, take the "structured change event set" and "metadata set of updated state field" as the processing object, and associate the corresponding "to-be-updated complete data lineage record" (i.e. the lineage record containing "flow path + transformation rule" generated before this metadata) through the "metadata unique ID". Second, update the "flow path node sequence" of the lineage record according to the change type: if the change type is "add": supplement the "initial flow node" at the "initial position" of the flow path (corresponding to the original area of the data factory), the node attributes include "node type = initial node", "change event identifier", "change timestamp" and "associated change reason log ID", forming the "lineage path supplemented with the initial node" (such as [original area - add initial node, bronze area node, silver area node...]); if the change type is "modify": locate the "transformation node related to the modified content" in the flow path (such as modifying the field mapping rule of the bronze area, then locating the bronze area transformation node), update the "transformation rule description" of the node (fill in the modified rule, such as "original field A→B, modified as A→C"), and add the "modification mark" attribute (including change event identifier and change reason log ID), forming the "lineage path updated with the transformation node"; if the change type is "delete": add the "termination flow node" at the "final position" of the flow path (corresponding to the consumption area of the data factory), the node attributes include "node type = termination node", "deletion timestamp", "change reason description" and "associated change reason log ID", forming the "lineage path marked with the termination node" (such as [original area node, bronze area node,..., consumption area - termination node]). Finally, perform "integrity check" on the updated lineage path (ensure that the node sequence is reasonable in time sequence and the associated log ID is valid), and generate a "complete data lineage record set updated with the path" through the back. This set realizes the dynamic linkage of the lineage path and the change behavior, and solves the defect that the traditional lineage only records the static path and cannot reflect the change history.

[0048] Preferably, the specific technical implementation of the change update unit synchronizing with the catalog generation module is as follows: first, taking the "metadata set of update state field", "complete data lineage record set of update path" and "structured change reason log set" as the processing objects, the association relationship among the three is established through the "metadata unique ID" and "change event identification", and a "metadata-change lineage association set" is formed. Secondly, the association set is synchronized to the "metadata governance list generated by the rule binding unit": the "updated data set state field" is synchronized to the "metadata state" field of the list; the "updated lineage flow path" is synchronized to the "associated lineage path" field of the list; the "core information of structured change reason log" (initiating subject, operation description, log ID) is synchronized to the "change traceability" field of the list; it is ensured that each entry in the metadata governance list reflects the latest change state, and an "updated metadata governance list" is generated. Finally, based on the "updated metadata governance list", a "unified metadata catalog" is constructed: the catalog is added with "search index" (based on metadata unique ID, data set state, business domain semantic label), which supports searching according to state, change time, lineage node and other dimensions; the "structured field" of the catalog is defined (including metadata basic information, latest state, complete lineage path, change traceability information, associated governance rule ID); the "access interface mapping" of the catalog is generated (associated with the metadata search interface of the subsequent interface configuration unit), which ensures that the data weaving environment can be accessed, and finally a "unified metadata catalog" is obtained.

[0049] Different from the traditional mode of "monitoring changes + manually updating metadata", the scheme is different in that: ① CDC-driven automatic closed loop: through log analysis CDC to realize non-intrusive change monitoring, avoid the influence of traditional trigger CDC on the performance of data factory, at the same time, automatically associate change reason log, state field, lineage path, form "monitoring-collection-update-synchronization" full-link automation, reduce intervention by more than 90% (general angle is "significantly reduce cost", specific example is from 100 times of operation per day to less than 10 times); ② Change-state-lineage three-dimensional linkage: strong binding of change type and dataset state (such as new → to be checked), strong association of state and lineage path (such as association of original area initial node in to-be-checked state), strong tracing of lineage path and change reason log (such as node containing log ID), solve the problem of "state and path disconnection, reason untraceable" in traditional change update; ③ Lineage dynamic evolution: instead of directly deleting the lineage path when deleting the change, the terminated node is marked, and the complete change history is preserved, providing support for "full life cycle tracing" in data weaving governance, and traditional solutions directly delete the path, resulting in loss of history.

[0050] Optionally, the interface configuration unit is specifically configured to configure a multi-dimensional metadata retrieval interface for the unified metadata catalog, and configure a governance rule execution engine, and the metadata retrieval interface and the governance rule execution engine are called by a data pipeline scheduling module and a data product access control module in a data weaving environment.

[0051] Optionally, the multi-dimensional metadata retrieval interface is used for responding to retrieval of the unified metadata catalog according to data domains, data ownership and dataset states, and the governance rule execution engine is used for automatically executing data domain access permission control and data product life cycle management in a metadata governance list.

[0052] Specifically, the interface configuration unit internally contains a multi-dimensional retrieval interface design module, a governance rule execution engine configuration module, and a module call adaptation module, which realize the specific process of "configuring multi-dimensional metadata retrieval interfaces and governance rule execution engines for unified metadata catalog, and supporting data pipeline scheduling modules and data product access control module calls in data weaving environments" as follows: Preferably, the specific implementation process of the multi-dimensional metadata retrieval interface design module is as follows: First, taking the "change update unit generated unified metadata catalog" as the processing object, parsing its structured fields (including metadata unique ID, business domain semantic label, data ownership label, data set state field, complete data lineage path, and associated governance rule ID), extracting the "retrieval key field" corresponding to the retrieval dimension - the "business domain semantic label" field corresponding to the data domain dimension, the "data ownership label" field corresponding to the data ownership dimension, and the "data set state field" corresponding to the data set state dimension, and generating a "catalog retrieval field mapping table" (recording the correspondence between retrieval dimensions and catalog fields, such as "data domain -> business domain semantic label"). Second, based on the "catalog retrieval field mapping table", the "request parameter structure" of the multi-dimensional metadata retrieval interface is designed: each retrieval dimension corresponds to an optional request parameter, and the parameter value supports exact matching (such as "data domain = retail domain") and fuzzy matching (such as "data ownership = e-commerce operation department"), and a "combined retrieval logic parameter" (such as "AND / OR" for multi-dimensional combined retrieval) is designed, and an "interface parameter definition table" (including parameter name, data type, retrieval dimension, and matching rule) is generated. Subsequently, taking the "interface parameter definition table" and "catalog retrieval field mapping table" as the processing object, the retrieval logic is realized: using the RESTful API style, after receiving the retrieval parameters transmitted by the client, the parameters are mapped to the retrieval key fields of the catalog; for single-dimensional retrieval, the corresponding field values are matched directly; for multi-dimensional combined retrieval, the logical operation between fields is performed according to the "combined retrieval logic parameter" (such as "data domain = retail domain AND data set state = to be verified"); at the same time, in order to improve the retrieval efficiency, a "multi-dimensional joint index" is constructed on the retrieval key fields of the unified metadata catalog (using the keyword type index of Elasticsearch, which supports fast matching), and a "multi-dimensional metadata retrieval interface instance" (including interface URL, request method, parameter example, and response format) is generated.Finally, the "multi-dimensional metadata retrieval interface instance" is functionally tested: simulate the retrieval request of the data weaving environment (such as "data domain = financial domain AND data ownership = risk control department"), verify whether the interface can return the metadata list meeting the conditions, record the retrieval response time (the general angle is "to meet the business query time requirement", the specific example is that the response time is less than or equal to 500 milliseconds), optimize the index structure to meet the response standard, and obtain the "available multi-dimensional metadata retrieval interface". The interface solves the problem of low efficiency of traditional single-dimensional retrieval and the inability to meet the multi-scene query requirements of the data weaving environment.

[0053] Preferably, the specific technical implementation of the governance rule execution engine configuration module is as follows: first, taking the "metadata governance list generated by the rule binding unit" and the "unified metadata catalog" as the processing object, the association between the two is established through the "metadata unique ID" - the "data domain access permission rule" (including the permission subject, the target data domain, the permission type, and the effective node) and the "data product life cycle management rule" (including the association relationship type, the management parameter of each stage, and the conversion trigger condition) are extracted from the metadata governance list, and these rules are bound with the corresponding metadata entries in the unified metadata catalog to generate a "metadata-rule association table" (recording the metadata unique ID, the rule type, the rule content, and the effective condition). Secondly, based on the "metadata-rule association table", the "core execution process" of the governance rule execution engine is designed: For data domain access permission rules: after receiving the "access request event" (including the request subject, target metadata ID, and access type), the engine is associated with the "metadata-rule association table" through the metadata ID to match the corresponding access permission rules, check whether the request subject meets the permission subject requirements and whether the access type is within the allowed range, and generate a "permission verification result" (pass / reject, with rejection reason); For data product life cycle management rules: the engine scans the "dataset status field" and "lineage flow node" of the unified metadata catalog regularly, matches the corresponding life cycle rules, judges whether the metadata meets the phase conversion conditions (such as "bronze zone data cleaning is completed and the quality verification pass rate is greater than or equal to the preset threshold") or archiving conditions (such as "the storage time of the original zone data is greater than or equal to 90 days"), and generates "life cycle management instructions" (phase conversion / archiving / preservation). Subsequently, the rule engine framework (such as Drools) is selected to integrate the above execution process, the "metadata-rule association table" is imported into the engine as a rule base, the "trigger mechanism" of the engine is configured (the permission verification trigger is "access request event driven", and the life cycle management trigger is "regular scanning + event driven"), and a "configured governance rule execution engine" is generated. Finally, an "execution result feedback mechanism" is designed for the "configured governance rule execution engine": the permission verification result and the life cycle management instruction are synchronized to the "governance execution log" field of the unified metadata catalog, and an independent execution log file (including execution time, rule ID, metadata ID, execution result, and exception information) is generated, ensuring that the data weaving environment can trace the rule execution process, and obtaining a "runnable governance rule execution engine". The engine is different from the traditional rule execution mode, realizes the automatic and real-time execution of governance rules, and reduces human operation bias.

[0054] Preferably, in a scenario, the module calls the specific implementation of the adaptation module as follows: First, with the "available multi-dimensional metadata retrieval interface" and the "executable governance rule execution engine" as the processing object, analyze the calling requirements of the "data pipeline scheduling module" and the "data product access control module" in the data weaving environment: the requirements of the data pipeline scheduling module: the multi-dimensional metadata retrieval interface needs to be called to query the "data set state (such as 'updated') of the to-be-scheduled metadata and the business domain", and the governance rule execution engine needs to be called to verify whether the scheduling operation conforms to the data domain access permission (such as 'whether the scheduling service has the right to read the golden area metadata'); the requirements of the data product access control module: the multi-dimensional metadata retrieval interface needs to be called to query the "data ownership of the to-be-accessed metadata", and the governance rule execution engine needs to be called to verify whether the user access request conforms to the permission rule (such as 'whether the retail domain user has the right to modify the retail domain metadata'). Based on the requirements, generate a "module calling requirement list" (including module name, calling component, calling scenario, request parameter, response requirement). Second, based on the "module calling requirement list", design the "calling adaptation interface": for the data pipeline scheduling module: provide a "metadata retrieval-permission verification" combined calling SDK (software development kit), which encapsulates the request parameters of the multi-dimensional metadata retrieval interface (such as "data set state = updated") and the permission verification parameters of the governance rule execution engine (such as "request subject = ETL scheduling service"), and supports synchronous calling process (first retrieval and then verification); for the data product access control module: provide a "permission verification-result feedback" independent calling API, which receives "user ID, metadata ID, access type" parameters, internally calls the governance rule execution engine, and supports associated calling of the multi-dimensional metadata retrieval interface to obtain "data ownership label" for auxiliary verification, and generates a "calling adaptation component" (including SDK, API document, parameter example). Subsequently, perform compatibility test on the "calling adaptation component": simulate the data pipeline scheduling module to send a request to query the updated retail domain metadata and verify the scheduling permission, verify whether the component can correctly forward to the multi-dimensional metadata retrieval interface and the governance rule execution engine, and return the "retrieval result + permission verification result"; simulate the data product access control module to send a "user access request", verify whether the component can return the "permission pass / reject" result and the associated metadata ownership information, solve the problems of calling parameter mismatch and response format incompatibility, and generate a "compatible calling adaptation component". Finally, deploy the "compatible calling adaptation component" to the "component registration center" of the data weaving environment, and provide a "calling configuration guide" (including registration address, parameter configuration, exception handling scheme), to ensure that the data pipeline scheduling module and the data product access control module can discover and call the multi-dimensional metadata retrieval interface and the governance rule execution engine through the registration center, forming a "interface-engine-module" calling closed loop.

[0055] Different from the traditional "single interface + manual call" configuration mode, the scheme is different in that: ① Multi-dimensional search linkage design: The three dimensions of data domain, data ownership and data set state are linked through joint indexing and logical operation search, solving the problem of traditional single-dimensional search "multiple queries, low efficiency", and the search response efficiency is improved by more than 50% (the general angle is "significantly improve the search efficiency", and the specific example is from 1 second to 0.5 seconds); ② Rule execution automation trigger: The governance rule execution engine adopts a "event-driven + timing scanning" dual-trigger mechanism, the permission check responds to access requests in real time, and the life cycle control actively scans the state, avoiding the lag caused by traditional "trigger rules"; ③ Modularized call adaptation: Special adaptation components (SDK + API) are designed for the differentiated needs of the two modules, rather than general interfaces, solving the problem of traditional adaptation "poor compatibility, complex call", reducing the difficulty of module integration.

[0056] Optionally, the change updating unit is further configured to: when synchronizing the updated data set state field and the flow path of the complete data lineage record to the metadata governance list, adjusting the effective range of the governance rule in the metadata governance list according to the data set change type.

[0057] Optionally, when the change updating unit is used to adjust the effective range of the governance rule in the metadata governance list according to the data set change type, a temporary governance rule is added for the newly added data, the effective period of which is a preset verification period, the parameters of the associated governance rule are updated for the modified data, including adjusting the access permission verification frequency, and the associated governance rule is marked as "invalid" for the deleted data.

[0058] Specifically, the change updating unit realizes the adjustment of the effective scope of the governance rule in the metadata governance list according to the data set change type through the rule association positioning module, the type-based rule adjustment module and the effective scope synchronization module. The specific process is as follows: preferably, the specific implementation process of the rule association positioning module is as follows: first, taking the "updated metadata governance list generated by the change updating unit" and the "structured change event set" as the processing objects, the two are associated through the "metadata unique ID" - the "change type" (addition / modification / deletion) and the corresponding "metadata unique ID" are extracted from the structured change event set, the "governance rule item" (including rule ID, rule type, original effective scope and associated lineage node) associated with the ID is extracted from the updated metadata governance list, and a "change-rule association temporary table" (fields include: metadata unique ID, change type, rule ID, rule type, original effective scope) is generated. Secondly, the "governance rule item" in the "change-rule association temporary table" is subjected to validity check: check whether the rule ID exists in the data weaving federation governance rule library, and whether the original effective scope matches the lineage node (such as whether the rule effective node is in the lineage path), and eliminate invalid rule items (such as rule ID does not exist), to obtain an "effective change-rule association table". Finally, an "adjustment priority" label is added to each record in the "effective change-rule association table": based on the influence degree of the change type on governance (deletion > modification > addition), high-priority adjustment is ensured to be executed first, and a "change-rule association table with priority" is generated. The table provides accurate rule positioning basis for subsequent type-based adjustment of rule effective scope, avoiding low efficiency caused by blind traversal of all rules. Preferably, the specific technical implementation of the type-based rule adjustment module for addition of data rules is as follows: first, taking the record with "change type = addition" in the "change-rule association table with priority" as the processing object, the corresponding "metadata unique ID" and "business domain semantic label" (obtained from the enhanced semantic metadata, such as "retail domain-user order") are extracted. Secondly, based on the "business domain semantic label", the "standard governance rule template of the same type of data" (such as the standard access permission rule and the life cycle rule of the retail domain user order data) is searched in the data weaving federation governance rule library, the "effective scope" field in the template is modified to "temporary effective", and a "temporary rule identifier" (such as rule ID suffix "-TEMP") is added, to generate a "temporary governance rule draft".Subsequently, configure the "effective period parameter" for the "draft temporary governance rule": the effective period is based on the "preset verification period" (generally set according to the importance of the data, for example, 24 hours for core data and 72 hours for non-core data), and the rule is added with "effective start time = change timestamp", "effective end time = change timestamp + preset verification period", and "expiration processing strategy = automatic submission of quality verification" (if the verification is passed, the rule is converted into a formal rule, and if the verification is not passed, intervention is triggered), to generate a "temporary governance rule with configured effective period". Finally, add the "temporary governance rule with configured effective period" to the "associated rule list" of the metadata governance list, mark it as "temporary effective" in the "rule status" field of the list, and associate the corresponding "change reason log ID" (explain the background of the addition of the temporary rule) to obtain a "governance list entry with added temporary rule". This temporary rule solves the "governance vacuum" problem of newly added data before the formal rule takes effect, ensuring that the data is controlled from the moment it is generated.

[0059] Preferably, in a scenario, the specific implementation of modifying data rule adjustment in the category rule adjustment module is as follows: First, take the record with "change type = modification" in the "priority change-rule association table" as the processing object, extract the corresponding "rule ID", "rule type" (data domain access permission rule / data product life cycle management rule) and "modification operation description" (obtained from the structured change reason log, such as "modify the user table mobile phone field format"). Second, analyze the impact range of "modification operation description" on the associated rule: if the rule type is "data domain access permission rule": if the modification content involves "field sensitivity improvement" (such as mobile phone field encryption method change), it is determined that the "access permission check frequency" needs to be adjusted (from low frequency to high frequency); if the rule type is "data product life cycle management rule": if the modification content involves "data cleaning logic change", it is determined that the "stage quality check threshold" needs to be adjusted (such as the null rate tolerance from 5% to 3%). Based on the impact analysis, generate a "rule parameter adjustment list" (containing rule ID, parameter name to be adjusted, original parameter value, new parameter value, adjustment basis). Then, update the parameters of the corresponding rule in the metadata governance list according to the "rule parameter adjustment list": for access permission check frequency: update the original "check once every hour" to "check once every 30 minutes", and record the values before and after the adjustment and the reason in the "parameter change log" field of the rule; for stage quality check threshold: update the original "null rate ≤ 5%" to "null rate ≤ 3%", and update the corresponding conversion trigger condition in the lineage flow node (such as the conversion from bronze area to silver area needs to meet the new threshold). Finally, perform "validity verification" on the updated rule parameters: simulate access request or life cycle conversion scenarios to verify whether the rule can be normally executed after parameter adjustment (such as whether high-frequency check triggers correct permission interception), and generate a "governance list entry after parameter update" if it passes. This adjustment realizes the dynamic adaptation of rule parameters and data modification content, avoiding the lack of governance precision caused by rule parameter solidification.

[0060] Preferably, the specific implementation process of deleting data rule adjustment in the sub-type rule adjustment module is as follows: first, taking the record with "change type = deletion" in the "priority change-rule association table" as the processing object, the corresponding "rule ID", "associated lineage node" (such as the consumption area termination node) and "deletion operation description" (such as "expiration 2023 credit data cleaning") are extracted. Secondly, all entries associated with the "rule ID" in the metadata governance list are located, the current state of the rule (such as "in effect" "temporary effect") is checked, and the "invalidation mark" (such as the state value is updated to "invalidation") is added to the "rule state" field of the list, while recording the "invalidation reason" (reference to the deletion operation description) and "invalidation timestamp" (consistent with the dataset deletion timestamp). Subsequently, "retroactive association identification" is added for the invalid rule: the rule ID is bound with the "termination flow node ID" in the "updated complete data lineage record", ensuring that the history of the rule is in effect (such as access records during the rule effective period, life cycle control records) can be queried through the lineage path, generating "invalid rule entry with retroactive identification". Finally, "invalid rule entry with retroactive identification" is retained in the metadata governance list (not physically deleted), and is only distinguished by "rule state = invalidation", while the complete information of the rule (including original effective range, parameters, invalidation reason) is added to the "historical rule archive" field of the list, obtaining "governance list entry marked invalidation". This design is different from the traditional way of directly deleting invalid rules, and supports the governance traceability of the whole life cycle of data through the preservation of historical records.

[0061] Preferably, the specific technical implementation of the effective range synchronization module is as follows: first, taking "governance list entries with added temporary rules", "governance list entries with updated parameters", and "governance list entries with marked invalid rules" as the processing objects, extract the "metadata unique ID", "rule ID", "adjusted rule state", and "effective range parameters" (such as the effective period of temporary rules, new parameters of modified rules, and invalid time of invalid rules) of each entry. Second, synchronize these adjustment information to the "updated metadata governance list": for new data scenarios, add temporary rule entries in the list and supplement the "temporary rule identifier" and "effective period" fields; for modified data scenarios, update the "parameter value" and "parameter change log" fields of the original rules in the list; for deleted data scenarios, update the "rule state", "invalid reason", and "trace association identifier" fields of the original rules in the list. During the synchronization process, "metadata unique ID + rule ID" is used to ensure that each adjustment record is accurately matched with the list entry, avoiding synchronization misplacement. Subsequently, generate "rule effective range adjustment log": record the adjustment timestamp, adjustment type (add temporary rule / update parameter / label invalid), involved metadata ID and rule ID, and effective range comparison before and after adjustment. This log is stored in association with the structured change reason log, forming a complete link of "change-adjustment-log". Finally, push the synchronized metadata governance list to the "unified metadata catalog" and update the content of the "associated governance rules" field in the catalog to ensure that the governance rule execution engine configured by the interface configuration unit can read the latest rule effective range, generating "unified metadata catalog with adjusted rule effective range".

[0062] Unlike the traditional "fixed rule effective range" or "manual adjustment" mode, the present scheme is different in that: ① scenario-based dynamic adjustment mechanism: design differentiated adjustment strategies for the three change types of addition, modification, and deletion (temporary rules with periods, parameters adapted with modifications, and invalid rules with traceability), solving the poor governance adaptability problem caused by the traditional "one-size-fits-all" rule; ② temporary rule life cycle closed loop: set a preset verification period and expiration processing strategy (automatic verification to official / interference) for the temporary rules of new data, avoiding the governance confusion caused by long-term effectiveness of temporary rules; ③ invalid rule traceability design: when deleting data, do not delete the associated rules, but mark them as invalid and bind the lineage termination node, realizing the full historical traceability of "data-rule-flow path", and the traditional solution directly deletes the rules, causing the traceability chain to break.

[0063] Optionally, the semantic mounting unit, when constructing the complete data lineage record, is further configured to: based on the data flow conversion logs between each processing area of the data factory, extract a set of structured metadata conversion rules including field mapping relationships and data cleaning logic, and embed the conversion rules into the corresponding flow conversion nodes of the complete data lineage record to form a complete data lineage record containing "flow path + conversion rule", and then store the complete data lineage record in association with the enhanced semantic metadata.

[0064] Specifically, the semantic mounting unit cooperates with a conversion log analysis module, a conversion rule extraction module, a rule-node embedding module, and an associated storage module to achieve the functions of "extracting conversion rules based on data flow conversion logs, embedding flow conversion nodes to form complete data lineage records, and storing in association with enhanced semantic metadata". The specific process is as follows: Preferably, the specific implementation process of the conversion log analysis module is as follows: First, taking "data flow conversion logs of each processing area (raw area, bronze area, silver area, gold area) of the data factory" as the processing object, the log collection range is determined to cover inter-area data transmission logs (such as "data import log" from raw area to bronze area), intra-area data processing logs (such as "field cleaning log" and "format conversion log" in bronze area), and cross-area rule execution logs (such as "aggregation calculation log" from silver area to gold area). The above logs are obtained in real time through a log collection interface (such as Flume's TailDir Source) to generate a "raw conversion log set" (each log contains log ID, source area, target area, processing timestamp, and log content). Second, the "raw conversion log set" is structured: the format of the log content (such as JSON, CSV, and unstructured text) is parsed, and "key elements" are extracted, including source data identifier (such as "user table_v1" in the raw area), target data identifier (such as "user table_cleaned" in the bronze area), processing operation type (such as "field mapping", "de-duplication", and "format conversion"), and processing tool identifier (such as "SparkSQL" and "Python Pandas"). The unstructured text log (such as SQL script) is converted into a "structured conversion log" in key-value pair format (fields include: log ID, source area, target area, source data identifier, target data identifier, processing operation type, processing tool identifier, original log content, and parsing timestamp). Finally, the "structured conversion log" is checked for completeness: it is checked whether there are missing key element logs (such as missing source data identifier or target data identifier), and the missing logs are marked as "to be completed" and triggered for secondary collection (such as re-pulling tool execution logs for the corresponding time period). The logs that pass the check generate a "set of structured conversion logs that pass the check". This set provides a standardized log data basis for subsequent conversion rule extraction, avoiding rule extraction deviation caused by chaotic log formats.

[0065] Preferably, the specific technical implementation of the conversion rule extraction module is as follows: first, taking the "set of structured conversion logs passed by verification" as the processing object, classifying it by "processing operation type" - dividing the logs into "field mapping type logs" (processing operation type contains "field mapping" "column conversion") and "data cleaning type logs" (processing operation type contains "de-duplication" "missing value processing" "format verification"), and generating a "classified conversion log set". Secondly, for "field mapping type logs", extract "field mapping relationship": If the processing tool identifier is "Spark SQL", parse the SELECT statement in the original log content, identify the mapping of "source field -> target field" (such as "user_name AS username" "age+1 AS age_plus"), record the mapping type (direct mapping / computational mapping) and the calculation logic (such as "age+1"); if the processing tool identifier is "Python Pandas", parse the DataFrame operation in the original log content (such as "df['username']=df['user_name'].str.strip()"), identify the source field (user_name), the target field (username) and the conversion function (strip()); integrate the extraction results into "structured field mapping rules" (fields include: rule ID, source data identifier, target data identifier, source field, target field, mapping type, calculation logic / conversion function, associated log ID). Subsequently, for "data cleaning type logs", extract "data cleaning logic": for "de-duplication" operation, extract the de-duplication dimension (such as "de-duplicate by user_id field"), the retention policy (such as "retain the latest record"); for "missing value processing", extract the processing field (such as "phone field"), the processing method (such as "fill in the default value 'unknown'" "delete missing records"); for "format verification", extract the verification field (such as "email field"), the verification rule (such as "regular expression ^[a-zA-Z0-9]+@[a-zA-Z0-9]+\.[a-zA-Z0-9]+$"); integrate the extraction results into "structured data cleaning rules" (fields include: rule ID, data identifier, processing field, cleaning operation type, cleaning logic, associated log ID). Finally, merge "structured field mapping rules" and "structured data cleaning rules" to generate "complete conversion rule set", and associate it with structured conversion logs through "associated log ID" to ensure that the rules can be traced back to the original logs.

[0066] Preferably, in a scenario, the specific implementation of the rule-node embedding module is as follows: First, take the "semantically mounted unit preliminary generated flow transfer path" (containing flow transfer nodes of each processing area, such as [original area node N1, bronze area node N2, silver area node N3]) and "complete conversion rule set" as the processing object, and establish the association between the flow transfer node and the conversion rule through the "source data identifier" and "target data identifier" - for example, the flow transfer from the original area node N1 to the bronze area node N2 corresponds to the conversion rule of "source data identifier = user table_v1, target data identifier = user table_cleaned", and a "node-rule association table" is generated (fields include: source node ID, target node ID, associated rule ID list). Secondly, add "conversion rule attributes" to each flow transfer node: for the "conversion node" between the source node and the target node (such as the intermediate conversion step from N1 to N2), embed the corresponding "structured field mapping rule" in the "field mapping rule" sub-attribute of the conversion node, and embed the "structured data cleaning rule" in the "data cleaning rule" sub-attribute; if the conversion node only involves a single operation (such as only field mapping or only cleaning), only the corresponding type of rule is embedded, and the sub-attribute of the rule type is marked as "none"; generate "flow transfer nodes embedded with conversion rules" (each node contains node ID, area identifier, source / target node association, field mapping rule, data cleaning rule). Subsequently, concatenate the "flow transfer nodes embedded with conversion rules" in the flow transfer order to form a "preliminary lineage record containing flow transfer path + conversion rule", where the path part maintains the node flow transfer order, and the rule part is bound to the corresponding conversion node. Finally, perform "rule-node matching verification" on the preliminary lineage record: check whether the rule of each conversion node matches the source / target data identifier of the node (such as whether the rule of N2 node corresponds to the conversion from N1 to N2), perform re-association on the rules that do not match (based on log ID to trace the correct node), and generate a "complete data lineage record" (containing node flow transfer sequence, each node conversion rule, rule association log ID) through the verification. This record is different from the traditional lineage containing only the path, and realizes the dual tracing of "flow transfer process + conversion details".

[0067] Preferably, the specific implementation process of the association storage module is as follows: first, taking the "complete data lineage record" and "enhanced semantic metadata generated by the semantic mounting unit" as the processing object, the "metadata unique ID" (the core identifier of the enhanced semantic metadata, such as "MD-001") of the two is extracted, and the association is established through the ID - the complete data lineage record adds the "associated metadata ID" field, and the enhanced semantic metadata adds the "associated lineage ID" field (the unique identifier of the lineage record, such as "LN-001"), and a "lineage-metadata association table" (fields include: metadata unique ID, lineage ID, association timestamp) is generated. Secondly, design the "association storage structure": use a graph database (such as Neo4j) to store the complete data lineage record, where the node corresponds to the flow node, and the edge attribute stores the conversion rule (field mapping, cleaning logic); use a relational database (such as MySQL) to store the enhanced semantic metadata, and establish a foreign key association with the lineage ID of the graph database through the "associated lineage ID" field; at the same time, add indexes (extracted from the enhanced semantic metadata) such as "business domain semantic label" and "data ownership label" to the lineage record in the graph database, and support querying the lineage record according to the semantic label. Subsequently, perform the association storage operation: write the "complete data lineage record" into the graph database, write the "enhanced semantic metadata" into the relational database, and synchronize the "lineage-metadata association table" to the association table of the two databases, to ensure that the lineage record can be directly queried through the metadata, and the semantic label of the metadata can be reversely queried through the lineage record. Finally, verify the effectiveness of the association storage: through the metadata unique ID query, check whether the corresponding complete data lineage record and the conversion rule therein can be returned; through the lineage ID query, check whether the corresponding enhanced semantic metadata and semantic label can be returned, repair the association relationship that fails to query (such as foreign key mismatch), and generate a "metadata-lineage set of completed association storage".

[0068] Different from the traditional "only record the data flow path" lineage implementation, the scheme is different in that: ① Conversion rule depth embedding: the field mapping relationship, data cleaning logic and flow node are bound, not stored independently, solving the problem of traditional lineage "path and rule disconnection, unable to trace the details of conversion"; ② Unified analysis of multi-type logs: Differentiated analysis logic is designed for the log design differences of different processing tools (Spark SQL, Pandas), ensuring accurate extraction of conversion rules in heterogeneous environments, and traditional solutions mostly support only single tool logs; ③ Semantic correlation storage: The lineage record and enhanced semantic metadata are associated through the metadata unique ID, and semantic tag index is added in the lineage, realizing the "semantic query→lineage tracing→rule verification" closed loop, and improving the operability of data governance.

[0069] Figure 2 A flowchart of an embodiment of the application for automatic metadata management and data weaving governance method. As shown in Figure 2 , it includes: based on the semantic model of the data weaving architecture, the structured metadata set of the data factory is mounted with semantic tags to generate enhanced semantic metadata and complete data lineage records; based on the data weaving federation governance rules, the enhanced semantic metadata and complete data lineage records are bound with governance rules to generate metadata governance lists bound with governance rules; based on the change data capture mechanism, when the data factory is monitored to have a data set change, the data set state field and the corresponding complete data lineage record flow path are updated to the metadata governance list to generate a unified metadata catalog; the unified metadata catalog is configured with a metadata retrieval interface and a governance rule execution engine, so that the unified metadata catalog is accessible in the data weaving environment.

[0070] The above only describes the preferred embodiments of the application and is not intended to limit the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An apparatus for automated metadata management and data weaving governance, comprising: The method comprises the following steps: a semantic mounting unit is used for mounting semantic labels on a structured metadata set of a data factory based on a semantic model of a data weaving architecture, to generate enhanced semantic metadata and complete data lineage records; a rule binding unit is used for binding governance rules to the enhanced semantic metadata and complete data lineage records based on data weaving federation governance rules, to generate metadata governance lists bound with the governance rules; a change updating unit is used for updating dataset state fields and corresponding complete data lineage records to the metadata governance lists based on a change data capture mechanism when a change in the data factory is monitored, to generate a unified metadata catalog; an interface configuration unit is used for configuring a metadata retrieval interface and a governance rule execution engine for the unified metadata catalog, to make the unified metadata catalog accessible in a data weaving environment.

2. The apparatus of claim 1, wherein, The semantic mounting unit is specifically used for mounting type-specific semantic labels on the structured metadata set of the data factory based on the semantic model of the data weaving architecture, and determining a flow path of the structured metadata set in the data factory, to generate the enhanced semantic metadata and complete data lineage records according to the semantic labels and the flow path.

3. The apparatus of claim 2, wherein, The structured metadata set comprises type-divided descriptive metadata, management metadata and structural metadata, and the semantic mounting unit mounts a business domain semantic label on the descriptive metadata, a data ownership label on the management metadata and a data association relationship label on the structural metadata when mounting type-specific semantic labels on the structured metadata set.

4. The apparatus of claim 2, wherein, The semantic model of the data weaving architecture comprises data domain association rules and knowledge graph entity relationships, and the semantic mounting unit determines the flow path of the structured metadata set in the data factory based on the knowledge graph entity relationships to preliminarily locate flow nodes of the structured metadata set in processing areas of the data factory, and then verifies the consistency of data domain attribution between the flow nodes according to the data domain association rules, and corrects the flow direction of the flow nodes with data domain attribution conflicts according to the data domain association rules to trace the flow path of the structured metadata set.

5. The apparatus of claim 1, wherein, The rule binding unit is specifically used for performing label quality and path integrity verification on the enhanced semantic metadata and complete data lineage records respectively, to obtain metadata and lineage records that pass the quality verification; and binding data weaving federation governance rules to the metadata and lineage records that pass the quality verification based on different scenes, to generate metadata governance lists bound with the governance rules. The rule binding unit is specifically used for performing label quality and path integrity verification on the enhanced semantic metadata and complete data lineage records respectively, to obtain metadata and lineage records that pass the quality verification; and binding data weaving federation governance rules to the metadata and lineage records that pass the quality verification based on different scenes, to generate metadata governance lists bound with the governance rules.

6. The apparatus of claim 5, wherein, The data weaving federal governance rule includes a data domain access permission rule and a data product life cycle management rule. When the rule binding unit binds the governance rules to the quality-verified metadata and lineage record in a scene, the data domain access permission rule is bound to the management metadata in the structured metadata set, and the data product life cycle management rule is bound to the structural metadata in the structured metadata set, to generate a metadata governance list bound with the governance rules.

7. The apparatus of claim 1, wherein, The change updating unit is specifically configured to: based on a change data capture mechanism, monitor a change type of a data set in a data factory in real time; when the change of the data set is monitored, synchronously collect a change reason log, and update a data set state field according to the change type, and update a flow path of a complete data lineage record, and synchronously update the updated field and the path to the metadata governance list, to generate a unified metadata catalog.

8. The apparatus of claim 7, wherein, The change type includes adding, modifying and deleting. When the change updating unit monitors the change of the data set, the change reason log including a change initiator and a change operation description is synchronously collected. When the data set state field is updated according to the change type, the corresponding "to be verified", "to be updated" and "to be archived" are added. When the flow path of the complete data lineage record is updated, the initial flow transfer node of the data set is added, the corresponding conversion node of the data set is updated, and the terminal node of the data set is marked. The updated data set state field and the flow path are synchronously updated to the metadata governance list, to generate the unified metadata catalog.

9. The apparatus of claim 1, wherein, The interface configuration unit is specifically configured to: configure a multi-dimensional metadata retrieval interface for the unified metadata catalog, and configure a governance rule execution engine. The metadata retrieval interface and the governance rule execution engine can be called by a data pipeline scheduling module and a data product access control module in a data weaving environment.

10. The apparatus of claim 9, wherein, The multi-dimensional metadata retrieval interface is used to respond to the retrieval of the unified metadata catalog according to a data domain, data ownership and data set state. The governance rule execution engine is used to automatically execute data domain access permission control and data product life cycle management in the metadata governance list.

Images