A metadata incremental synchronization and change tracking method, system and storage medium

Abstract

This application discloses a method, system, and storage medium for incremental metadata synchronization and change tracking. The method includes: acquiring the current state information of a metadata object; generating a first fingerprint representing the technical form and a second fingerprint representing the business semantics; comparing the fingerprints of the current state with those of the baseline state; determining the change type based on the comparison result and driving the update of the three-state flag; invoking the corresponding processing strategy according to the update type flag to synchronize and generate change tracking records. This application accurately distinguishes between structural changes and semantic changes through multi-dimensional fingerprints, achieves seamless state connection and preserves historical information through three-state flags, invokes differentiated processing strategies according to the change type, and achieves full lifecycle traceability through change tracking records. It solves the problems of coarse change detection, easy overwriting of governance results, reliance on manual decision-making, and difficulty in historical tracing in traditional metadata synchronization.

Description

Technical Field

[0001] This invention belongs to the field of data governance and metadata management technology, and in particular relates to a method, system and storage medium for incremental synchronization and change tracking of metadata. Background Technology

[0002] In the field of data governance and metadata management, enterprise-level data governance platforms need to continuously synchronize metadata information from various heterogeneous data sources (such as MySQL, Oracle, Hive, etc.) to support business scenarios such as data asset management, data census, and data compliance auditing. Currently, the mainstream metadata synchronization technologies mainly include two types of solutions: full-coverage synchronization and incremental synchronization based on simple comparison. Full-coverage synchronization pulls all metadata from the source end periodically and directly overwrites the target platform. Although simple to implement, it suffers from serious efficiency problems and unconditionally overwrites high-value business semantic information (such as field comments) already supplemented by manual or intelligent algorithms in the target platform, resulting in the loss of previous governance results. On the other hand, incremental synchronization solutions based on simple comparison usually use overall hash comparison or rely on the source end's timestamp to determine changes. The former can only detect whether the object has changed, but cannot distinguish whether it is a change in technical attributes (such as data type, field length) or a change in business semantics (such as comment modification). The latter is limited by the fact that most database metadata tables lack reliable timestamp fields, which limits the accuracy and applicability of incremental detection. More importantly, existing technologies generally lack the ability to finely identify change types and intelligent merging decision-making mechanisms. They cannot implement differentiated processing strategies for different types of changes during the synchronization process, nor can they protect the high-quality business annotations intelligently completed by the governance platform from being covered by null values ​​or meaningless information from the source. At the same time, existing solutions usually only record synchronization task logs, lacking the ability to accurately mark the atomic states of each metadata object such as "addition", "logical deletion", and "update" and trace the entire lifecycle. This results in a vague metadata change history and an untraceable evolution process, making it difficult to meet the deep management needs of data asset operation, compliance auditing, and root cause analysis. Summary of the Invention

[0003] (a) Purpose of the invention

[0004] To overcome the above shortcomings, the present invention aims to provide a method, system, and storage medium for incremental metadata synchronization and change tracking, in order to solve the four major defects of existing metadata synchronization technologies: coarse change detection granularity, easy overwriting of intelligent governance results, lack of intelligent basis for synchronization decisions, and fuzzy object state management, which prevent the realization of refined, intelligent, and traceable incremental metadata synchronization and change tracking.

[0005] (II) Technical Solution

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] A method for incremental synchronization and change tracking of metadata based on smart fingerprints includes the following steps:

[0008] Retrieve the current state information of metadata objects in the data source;

[0009] For metadata objects, generate a first fingerprint to represent their technical form and a second fingerprint to represent their business semantics.

[0010] The first and second fingerprints corresponding to the current state information are compared with the first and second fingerprints corresponding to the pre-stored baseline state information.

[0011] Based on the comparison results, determine the change type of the metadata object and drive the update of the three-state flags used to indicate the object's status. The three-state flags include at least an addition flag, a deletion flag, and an update type flag.

[0012] Based on the change type indicated by the update type flag, the corresponding processing strategy is invoked to synchronize the metadata object and generate the corresponding change tracking record.

[0013] This application constructs a multi-dimensional composite fingerprint system by generating a first fingerprint representing the technical form and a second fingerprint representing the business semantics of metadata objects. Compared with the traditional overall hashing method, which can only determine whether an object has changed, this method can accurately distinguish between structural changes and semantic changes, providing a refined perception foundation for subsequent differentiated processing. By mapping the fingerprint comparison results to a three-state flag system consisting of a new flag, a deletion flag, and an update type flag, a seamless connection from technical perception to business status is achieved. This not only accurately identifies the change type of the object but also completely preserves the historical information of the object through a logical deletion mechanism, avoiding historical traceability gaps caused by physical deletion. On this basis, the corresponding processing strategy is invoked according to the change type indicated by the update type flag, realizing adaptive synchronous triggering based on change type. This allows structural changes and semantic changes to enter differentiated processing channels, thereby providing a decision-making basis for the protection of semantic assets while ensuring technical consistency. By generating change tracking records containing information such as object identifier, change type, merging decision, and before and after snapshots during the synchronous processing, a complete atomic archive is established for each state change, realizing full lifecycle traceability of metadata objects. This method fundamentally solves the technical problems of coarse change detection granularity, easy overwriting of governance results, reliance on manual decision-making, and difficulty in historical tracing in traditional metadata synchronization through a complete closed loop of "fingerprint perception - flag-driven - policy diversion - tracking and retention".

[0014] In some embodiments,

[0015] The first fingerprint is a structural fingerprint, which is generated by extracting the technical attributes of the metadata object and performing hash calculations.

[0016] The second fingerprint is a semantic fingerprint, which is generated by extracting the business attribute text of the metadata object and converting it into a numerical vector using a natural language processing model.

[0017] This embodiment achieves a separate representation of technical form and business meaning by constructing a composite fingerprint of structural and semantic dimensions for metadata objects. Compared with the traditional overall hashing method, which can only determine whether an object has changed, this embodiment can accurately distinguish between structural changes and semantic changes, providing a fine-grained perception foundation for subsequent differentiated processing and effectively solving the problem of coarse granularity in change detection.

[0018] In some embodiments,

[0019] Technical attributes include one or more of the following: data type, data length, whether it is nullable, and whether it is a primary key.

[0020] Business attribute text includes field comments.

[0021] In some embodiments, the step of determining the change type of the metadata object based on the comparison result and driving the update of the tri-state flag used to indicate the object's state specifically includes:

[0022] If the metadata object exists in the current status information but not in the baseline status information, then the addition flag is set to valid to indicate that the metadata object is a newly added object;

[0023] If the metadata object exists in the baseline status information but not in the current status information, then the deletion flag is set to valid to indicate that the metadata object is a logically deleted object;

[0024] If the metadata object exists in both the current state information and the baseline state information, then:

[0025] When the comparison result shows that the first fingerprint is different and the second fingerprint is the same, set the update type flag to indicate structural change;

[0026] When the comparison result shows that the first fingerprint is the same but the second fingerprint is different, set the update type flag to indicate semantic change;

[0027] When the comparison result shows that the first fingerprint is different and the second fingerprint is different, set the update type flag to indicate mixed change.

[0028] This embodiment constructs a complete vector describing the synchronization state of an object by precisely mapping the multi-dimensional fingerprint comparison results to three flag bits: addition, logical deletion, and update type. It achieves seamless integration from technical awareness to business status, not only accurately identifying the change type of an object but also completely preserving the object's historical information through a logical deletion mechanism. This avoids historical tracing gaps caused by physical deletion, upgrading metadata management from extensive synchronization to refined state management.

[0029] In some embodiments, the step of invoking the corresponding processing strategy to synchronize the metadata object according to the change type indicated by the update type flag specifically includes:

[0030] When the update type flag indicates a structural change or a mixed change, the technical attributes of the metadata object are updated to the technical attributes in the current status information through the structural update channel.

[0031] When the update type flag indicates a semantic change or a hybrid change, the business attributes of the metadata objects in the baseline state information are evaluated through the semantic decision channel, and a decision is made on whether to update the business attributes to the business attributes in the current state information based on the evaluation results.

[0032] Structural and hybrid changes involve alterations to technical attributes that may affect downstream data connectivity and processing logic. Therefore, a dedicated channel is used to ensure timely synchronization. Semantic changes, on the other hand, are assessed through an intelligent decision-making channel before being processed, thus providing a decision-making basis for the protection of semantic assets while ensuring technical consistency.

[0033] In some embodiments, the step of evaluating the value of business attributes of metadata objects in baseline state information and deciding whether to update the business attributes to the business attributes in the current state information based on the evaluation results specifically includes:

[0034] If the business attributes in the baseline status information are assessed as high value, then the business attributes in the baseline status information are retained, and the business attributes in the current status information are ignored.

[0035] If a business attribute in the baseline status information is assessed as low value, the business attribute will be updated to the business attribute in the current status information.

[0036] This implementation introduces a business semantic value assessment mechanism to determine the value of existing annotations on the platform. For high-value annotations generated by the platform's intelligent completion algorithm or confirmed by business experts, the system automatically retains and ignores low-quality information from the source. For low-value annotations synchronized from the source that are empty, contain meaningless placeholders, or contain garbled characters, the new information from the source is adopted. This mechanism fundamentally solves the industry pain point that intelligent governance results are easily overwritten during the synchronization process, achieving zero-loss preservation of high-value semantic assets.

[0037] In one specific implementation, metadata objects are stored on the platform with attached attribute tags. If a business attribute is generated by the platform's intelligent completion algorithm, the system automatically marks its source as "AI_GENERATED"; if it is manually reviewed and confirmed by business experts, it is marked as "VERIFIED". The value evaluator determines its value by reading this tag: when the tag is "AI_GENERATED" or "VERIFIED", the business attribute is evaluated as high-value; otherwise, it is treated as low-value.

[0038] In some embodiments, the change tracking record includes at least the following information: a unique identifier of the metadata object, a synchronization batch number, a change type, a summary of the change content, a merge decision, a snapshot of the key status before the change, a snapshot of the key status after the change, and a synchronization timestamp.

[0039] This embodiment provides full lifecycle traceability capabilities, supporting historical queries and version comparisons by object, time range, and change type. It can meet the in-depth management needs of data asset operation, compliance audit, and root cause analysis, and shorten the audit report generation time from the traditional days to minutes.

[0040] Another aspect of this application provides a metadata incremental synchronization and change tracking system based on intelligent fingerprints, including:

[0041] The data acquisition module is used to obtain the current status information of metadata objects in the data source;

[0042] The fingerprint generation module is used to generate a first fingerprint that represents the technical form of a metadata object and a second fingerprint that represents its business semantics.

[0043] The comparison module is used to compare the first fingerprint and the second fingerprint corresponding to the current state information with the first fingerprint and the second fingerprint corresponding to the pre-stored baseline state information.

[0044] The status management module is used to determine the change type of the metadata object based on the comparison results and drive the update of the three-state flags that indicate the object status. The three-state flags include at least an addition flag, a deletion flag, and an update type flag.

[0045] The synchronization processing module is used to invoke the corresponding processing strategy according to the change type indicated by the update type flag, so as to synchronize the metadata object and generate the corresponding change tracking record.

[0046] This system, through modular design, decouples core functions such as fingerprint generation, comparison, status management, and synchronization processing, enabling the complete deployment of the aforementioned methods within a computer system. Each module has clearly defined responsibilities and collaborates efficiently, supporting fully automated processing from fingerprint generation to change tracking. This provides data governance platforms with a directly integrable metadata synchronization solution possessing intelligent decision-making capabilities.

[0047] In some embodiments, the first fingerprint is a structural fingerprint, generated by extracting the technical attributes of the metadata object and performing hash calculations; the second fingerprint is a semantic fingerprint, generated by extracting the business attribute text of the metadata object and converting it into a numerical vector using a natural language processing model.

[0048] In another aspect, this application provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the above-described method.

[0049] This storage medium enables the method of this invention to be deployed and distributed as a software product. It facilitates migration and reuse across different data governance platforms, lowers the barrier to implementation of the technical solution, and expands the application scenarios of this invention. Attached Figure Description

[0050] Figure 1 This is an overall flowchart of the metadata incremental synchronization and change tracking method based on intelligent fingerprint of the present invention;

[0051] Figure 2 This is a schematic diagram of the multidimensional intelligent fingerprint generation process of the present invention;

[0052] Figure 3 This is a flowchart of the change detection, three-state flag setting, and intelligent merging decision-making logic of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0054] This invention provides a method for incremental synchronization and change tracking of metadata based on intelligent fingerprints. First, the system acquires the current state information of metadata objects in the data source. These metadata objects can be basic units of data assets such as data tables and fields. The current state information includes the object's technical attributes, such as data type and field length, and its business attributes, such as field comments. Further, for each metadata object, the system generates a first fingerprint representing its technical form and a second fingerprint representing its business semantics. Specifically, the first fingerprint is a fixed-length string obtained by hashing the technical attributes after standardization, while the second fingerprint is generated by converting the business attribute text into a fixed-dimensional numerical vector using a natural language processing model (e.g., using the Sentence-BERT model to convert the field comment text into a 768-dimensional numerical vector as a semantic fingerprint). Based on this, the system compares the first and second fingerprints corresponding to the current state information with the first and second fingerprints corresponding to the pre-stored baseline state information dimension by dimension. This baseline state information is a snapshot of the metadata object's state persistently stored during the last synchronization, containing the object's historical fingerprint information. Based on the comparison results, the system determines the change type of the metadata object and drives the update of the three-state flags indicating the object's state. These three-state flags include at least an addition flag, a deletion flag, and an update type flag. It's important to note that the updates of the three-state flags are not independent but are precisely mapped based on the fingerprint comparison results. Finally, based on the change type indicated by the update type flag, the system invokes the corresponding processing strategy to synchronize the metadata object, generating corresponding change tracking records in the process. In this way, through fingerprint comparison, state flag-driven processing, and differentiated strategy invocation, fine-grained control and full traceability of incremental metadata synchronization are achieved. It should be noted that the three-state flag is a combined state vector composed of the addition flag, deletion flag, and update type flag, used to uniquely describe the state of the metadata object during the synchronization process. Furthermore, the addition flag and deletion flag are mutually exclusive, and the update type flag is only valid if the object exists; together, the three constitute a complete description of the object's synchronization state.

[0055] like Figure 1As shown, the overall process of the metadata incremental synchronization and change tracking method provided by this invention starts from step S10, which generates intelligent fingerprints. This step generates a structural fingerprint for each metadata object to represent its technical form and a semantic fingerprint to represent its business semantics. Then, step S20 determines the change status through fingerprint comparison and sets three-state flags accordingly, including an addition flag, a deletion flag, and an update type flag. Based on this, step S30 performs traffic splitting based on the value of the update type flag: objects with structural or mixed changes are assigned to a high-priority processing channel to ensure timely synchronization; objects with semantic changes are transferred to an intelligent decision-making channel for value assessment before processing; other objects without substantial changes are processed using a batch asynchronous scheduling method to optimize system resource utilization. After traffic splitting, step S40 performs an intelligent merging operation, where technical attributes unconditionally adopt the latest value from the source, while semantic attributes are updated based on a value-priority principle. Step S50 records change tracking information, including not only changes in the three-state flags but also snapshots of key states before and after the change and the reasons for the merging decision. Finally, step S60 updates the asset status and triggers impact analysis, forming a management closed loop from the start of synchronization to asset status synchronization. This flowchart clearly illustrates the complete technical path from fingerprint generation to asset update, embodying the intelligent closed-loop design concept of "perception—decision—execution—recording—feedback".

[0056] Building upon the methods described above, this embodiment further defines the fingerprint generation method. Specifically, the first fingerprint uses a structural fingerprint, generated as follows: The system extracts the technical attributes of the metadata object, including data type, data length, whether it is empty, and whether it is a primary key. After standardizing these technical attributes according to preset rules, they are input into a cryptographic hash function such as MD5 or SHA-256 to calculate a fixed-length hash value, which is the structural fingerprint. This structural fingerprint uniquely represents the technical form of the object; any change in the technical attributes will alter the structural fingerprint. Correspondingly, the second fingerprint uses a semantic fingerprint, generated as follows: The system extracts the business attribute text of the metadata object, typically field annotations. This text is input into a lightweight natural language processing model, such as Sentence-BERT or a word vector model. The model converts the text into a fixed-dimensional numerical vector, which is the semantic fingerprint. This semantic fingerprint captures the business meaning of the object; similar annotations have a close distance in the vector space. In this way, the structural fingerprint and semantic fingerprint characterize the essential features of the metadata object from both technical and business dimensions, providing a foundation for subsequent refined change detection.

[0057] like Figure 2As shown, the multidimensional smart fingerprint generation process has a clear parallel processing architecture. The system receives the raw metadata object, which contains a complete set of attributes for the table or field. During the fingerprint generation phase, the system starts two independent processing pipelines in parallel.

[0058] The first pipeline is responsible for generating structural fingerprints: First, it extracts structural attributes from the original metadata, including key information such as data type, data length, whether it is empty, and whether it is a primary key. This information is then sent to the structural standardization processor for unified format processing, such as removing redundant spaces and sorting attributes. Finally, the standardized attribute string is input into a hash function to generate a fixed-length hash value as the structural fingerprint.

[0059] The second pipeline is responsible for semantic fingerprint generation: extracting semantic attributes from the original metadata, mainly including field annotations and business tags, feeding this text information into a lightweight NLP model for text vectorization, outputting the original vector, and then performing dimension normalization and L2 regularization processing through the semantic vector standardization module to finally generate a standardized semantic fingerprint.

[0060] After parallel processing by two pipelines, the structural fingerprint and semantic fingerprint are packaged into a composite fingerprint object, which is then stored in the baseline storage and associated with the metadata object ID, storage timestamp, and version information. This parallel processing architecture fully demonstrates the efficient design of this invention in the fingerprint generation stage, laying a solid data foundation for subsequent incremental synchronization.

[0061] In practical implementation, the aforementioned technical attributes may include one or more of the following: data type, data length, nullability, and primary key. Preferably, the system can collect all of the above technical attributes simultaneously to construct a complete structural fingerprint. For example, for a field in a MySQL database, its technical attributes may include VARCHAR type, length 50, nullability allowed, and non-primary key. The system concatenates this information into a standardized string and then performs hash processing. It is worth noting that the types of technical attributes can be adapted according to the data source type. Different database systems may support different sets of attributes, and the system extracts and standardizes them uniformly through the adapter pattern. The business attribute text specifically includes field comments, which can be descriptive text manually added by the database administrator or business meaning explanations intelligently completed by the data governance platform using AI algorithms. Preferably, the system preprocesses the field comments during extraction, removing meaningless spaces, line breaks, and other interfering information to ensure the stability and accuracy of the semantic fingerprint.

[0062] The following is a detailed description of how to determine the change type based on the fingerprint comparison results and drive the tri-state flag update:

[0063] Specifically, the system first performs object-level change detection: If a metadata object exists in the current status information but not in the baseline status information, the object is determined to be a newly added object. The system sets the add flag to valid, the delete flag to invalid, and the update type flag to empty in the record. If a metadata object exists in the baseline status information but not in the current status information, the object is determined to be logically deleted. The system does not physically delete the record but instead sets its delete flag to valid, its add flag to invalid, and its update type flag to empty, thus preserving the object's historical information. For metadata objects that exist in both the current and baseline status information, the system performs attribute-level change classification. The specific comparison rules are as follows:

[0064] When the comparison result shows that the first fingerprint is different but the second fingerprint is the same, it indicates that only the technical form has changed but the business meaning has not changed. In this case, the system sets the update type flag to indicate structural change. When the comparison result shows that the first fingerprint is the same but the second fingerprint is different, it indicates that only the business semantics have changed but the technical form has not changed. In this case, the update type flag is set to indicate semantic change.

[0065] When the comparison results show that both the first and second fingerprints are different, it indicates that changes have occurred in both technical and business aspects. In this case, the update type flag is set to indicate a mixed change. In this way, the system refines the change type into three atomic states—structural change, semantic change, and mixed change—through precise fingerprint comparison and three-state flag settings, providing a precise decision-making basis for subsequent differentiated processing.

[0066] like Figure 3 As shown, the change detection and intelligent merging decision logic of this invention forms a complete closed-loop processing flow. After the process begins, the system acquires the metadata to be synchronized, and then iterates through all objects for processing one by one. For each object, the system first determines its existence in the current state information and the baseline state information. If it exists in the current state information but not in the baseline state information, it is determined to be a newly added object, the system sets the addition flag to valid and records it in the tracking record table. If it exists in the baseline state information but not in the current state information, it is determined to be a deleted object, the system sets the deletion flag to valid and records it in the tracking record table, while fully retaining the object's historical information without physical deletion. If the object exists in both the current state information and the baseline state information, it enters the fingerprint comparison stage. By comparing the differences between the structural fingerprint and the semantic fingerprint, the system classifies the object into three cases:

[0067] If only the structural fingerprint changes, it is determined to be a structural change, and the system synchronizes the technical attributes and records them in the tracking record table;

[0068] If only the semantic fingerprint changes, it is determined to be a semantic change. The system then enters the value assessment process, decides whether to update the semantic attributes based on the assessment results, and records the decision process in the tracking record table.

[0069] If both the structural fingerprint and semantic fingerprint change, it is considered a mixed change. The system simultaneously executes two branches: technical attribute synchronization and value assessment. Technical attributes are unconditionally synchronized from the source, while semantic attributes are updated based on the value assessment results. Finally, the complete processing results are recorded in the tracking record table. After all objects have been processed, the system updates the asset status and ends the synchronization process.

[0070] Figure 3 The invention clearly demonstrates the complete decision-making chain from adding / deleting, fingerprint comparison, change classification to intelligent merging decision-making, reflecting the sophisticated design of the invention in the change detection and processing stage.

[0071] The following explains how to invoke the corresponding processing strategy based on the update type flag:

[0072] Specifically, when the update type flag indicates a structural change or a hybrid change, the system synchronizes metadata objects through the structural update channel. This channel uses a high-priority processing queue to ensure that synchronization tasks involving changes in technical attributes are executed promptly, as changes in technical attributes may affect downstream data connections, storage, and processing logic, requiring rapid synchronization to ensure the normal operation of the data platform. In this channel, the system unconditionally updates the technical attributes of the metadata object to the technical attributes in the current state information, including key parameters such as data type and length. On the other hand, when the update type flag indicates a semantic change or a hybrid change, the system processes it through the semantic decision channel. This channel does not directly update business attributes, but first evaluates the value of the business attributes of the metadata object in the baseline state information, and decides whether to update the business attributes to the business attributes in the current state information based on the evaluation results. In particular, for hybrid change objects that include both structural and semantic changes, the system triggers both the structural update channel and the semantic decision channel simultaneously, but their execution is parallel and independent: technical attributes are unconditionally updated from the source, while business attributes require value evaluation before a decision is made on whether to update them. In this way, adaptive scheduling based on change type is achieved, which not only ensures the timeliness of technical consistency, but also reserves decision-making space for the protection of semantic assets.

[0073] The following is a detailed description of semantic value assessment and intelligent merging decision-making:

[0074] The system incorporates a business attribute value evaluator, which determines the value of business attributes of metadata objects in the baseline status information. The evaluation rules are as follows: if a business attribute is generated by the platform's intelligent completion algorithm or manually confirmed by business experts, and its content is rich and accurately described, it is evaluated as high-value; if a business attribute is synchronized from the source and its content consists of empty strings, meaningless placeholders such as "col1" or "field 1," or contains garbled information, it is evaluated as low-value. Based on the above value evaluation results, the system executes an intelligent merging decision. When a business attribute is evaluated as high-value, the system retains the business attribute in the baseline status information, completely ignoring the business attribute in the current status information, and records the reason for the "semantic preservation" decision for subsequent auditing and traceability. When a business attribute is evaluated as low-value, the system updates the business attribute to the business attribute in the current status information, adopting the semantic information provided by the source. It is worth noting that this decision logic also applies to mixed change scenarios: when a high-value annotation of a mixed change object is preserved, the system will adjust the update type flag from "mixed change" to "structural change only" to accurately reflect the actual operation performed. In this way, by prioritizing semantic value in the intelligent merging mechanism, the industry pain point that the intelligent completion results of the governance platform are easily covered during the synchronization process is fundamentally solved.

[0075] The following is a detailed description of the specific components of the change tracking record:

[0076] A change tracking record is generated for each object whose state changes using atomic operations. This record is stored in a structured format and includes the following information: a unique identifier for the metadata object, used to associate it with a specific table or field; a synchronization batch number, a unique identifier for this synchronization task, facilitating batch queries; a change type, combining addition, deletion, and update flags, with possible values ​​including addition, logical deletion, structural change, semantic change, and mixed change; a change summary, describing the specific content of the change in readable text, such as "the length of the 'phone' field has been changed from 20 to 50"; a merge decision, recording the reasons for the decision made during the intelligent merging process, such as "adopting technical attributes, retaining the original annotation due to its high value"; snapshots of key states before and after the change, both stored in JSON format, fully recording the core attributes of the metadata object before and after the change; and a synchronization timestamp, accurate to the millisecond level, recording the time the change occurred. Preferably, the system can retain all historical tracking records for logically deleted objects, forming a complete lifecycle archive from creation to logical deletion. Based on these tracking records, the system provides a powerful historical query interface, supporting searches by object, time range, and change type, and allowing comparison of differences between any two snapshots. This enables full traceability of metadata changes, meeting the in-depth management needs of data asset operation, compliance auditing, and root cause analysis.

[0077] This embodiment also provides a metadata incremental synchronization and change tracking system based on intelligent fingerprints, which adopts a modular architecture design. Specifically, the system includes a data acquisition module for obtaining the current state information of metadata objects from various data sources such as MySQL, Oracle, and Hive. This module supports multiple database connection methods and can adapt to different data source types. The fingerprint generation module receives the raw metadata output by the data acquisition module and generates a first fingerprint representing its technical form and a second fingerprint representing its business semantics for each metadata object. Preferably, the fingerprint generation module adopts a parallel processing pipeline to simultaneously process the hash calculation of the structural fingerprint and the vectorization conversion of the semantic fingerprint, thereby improving processing efficiency. The comparison module compares the first and second fingerprints corresponding to the current state information with the first and second fingerprints corresponding to the pre-stored baseline state information. This module maintains a baseline database that stores a complete state snapshot from the last synchronization. The state management module receives the output results from the comparison module, determines the change type of the metadata object based on the comparison results, and drives the update of the three-state flags: addition flag, deletion flag, and update type flag. The synchronization processing module invokes the corresponding processing strategy based on the change type indicated by the update type flag to synchronize the metadata object, generating corresponding change tracking records in the process. The modules communicate through standardized interfaces, forming a complete processing chain of "fingerprint generation—fingerprint comparison—state-driven—policy invocation—tracking retention." In this way, the system achieves complete deployment of the aforementioned methods in a computer system, providing a directly integrateable metadata synchronization solution with intelligent decision-making capabilities for data governance platforms.

[0078] In a preferred implementation of this system, the first fingerprint specifically adopts a structural fingerprint, which is generated as follows: the fingerprint generation module extracts the technical attributes of the metadata object, including one or more of the following: data type, data length, whether it is nullable, and whether it is a primary key. After standardizing these attributes, they are input into a hash function such as MD5 or SHA-256 to calculate and generate a fixed-length structural fingerprint. The second fingerprint specifically adopts a semantic fingerprint, which is generated as follows: the fingerprint generation module extracts the business attribute text of the metadata object, especially field comments. This text is input into a natural language processing model such as Sentence-BERT or word vectors, and the model outputs a fixed-dimensional numerical vector as the semantic fingerprint. It is worth noting that the generation processes of the structural fingerprint and the semantic fingerprint are parallel and independent, without interfering with each other. Preferably, the system can be configured with different hash algorithms and natural language processing models to adapt to different application scenarios and performance requirements. For example, in scenarios with high security requirements, the SHA-256 algorithm can be selected instead of MD5; in scenarios with high requirements for semantic similarity accuracy, the Sentence-BERT model can be selected instead of a simple word vector model. In this way, the fingerprint generation module of the system has good configurability and scalability.

[0079] This application also provides a computer-readable storage medium, which can be a hard disk, solid-state drive, USB flash drive, optical disk, or any other medium with data storage function. The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7. Specifically, when the storage medium is installed or connected to a computer device, the processor reads the program instructions in the medium and executes operations such as metadata acquisition, fingerprint generation, fingerprint comparison, three-state flag update, policy invocation, and tracking record generation according to the instruction sequence. Preferably, the storage medium can be distributed and deployed as a carrier of an independent software product. Users only need to install the program in the medium onto a server or cloud host to quickly deploy the technical solution of this invention. It is worth noting that the storage medium does not depend on a specific operating system or hardware platform and can run on mainstream operating systems such as Windows and Linux, as well as on processors with different architectures such as x86 and ARM. In this way, the technical solution of this invention can be migrated and reused between different data governance platforms in the form of a software product, lowering the threshold for implementation and expanding the application scenarios of this invention.

[0080] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A method for incremental synchronization and change tracking of metadata based on intelligent fingerprints, characterized in that, Includes the following steps: Retrieve the current state information of metadata objects in the data source; For the metadata object, a first fingerprint is generated to represent its technical form and a second fingerprint is generated to represent its business semantics. The first and second fingerprints corresponding to the current state information are compared with the first and second fingerprints corresponding to the pre-stored baseline state information. Based on the comparison results, the change type of the metadata object is determined, and the three-state flag used to indicate the object's state is updated. The three-state flag includes at least an addition flag, a deletion flag, and an update type flag. Based on the change type indicated by the update type flag, the corresponding processing strategy is invoked to synchronize the metadata object and generate corresponding change tracking records.

2. The method according to claim 1, characterized in that, The first fingerprint is a structural fingerprint, generated by extracting the technical attributes of the metadata object and performing hash calculations; The second fingerprint is a semantic fingerprint, which is generated by extracting the business attribute text of the metadata object and converting it into a numerical vector using a natural language processing model.

3. The method according to claim 2, characterized in that, The technical attributes include one or more of the following: data type, data length, whether it is nullable, and whether it is a primary key; The business attribute text includes field comments.

4. The method according to claim 1, characterized in that, The step of determining the change type of the metadata object based on the comparison results and driving the update of the tri-state flag used to indicate the object's state specifically includes: If the metadata object exists in the current status information but not in the baseline status information, then the addition flag is set to valid to indicate that the metadata object is a newly added object; If the metadata object exists in the baseline status information but not in the current status information, then the deletion flag is set to valid to indicate that the metadata object is a logically deleted object; If the metadata object exists in both the current state information and the baseline state information, then: When the comparison result shows that the first fingerprint is different and the second fingerprint is the same, the update type flag is set to indicate a structural change; When the comparison result shows that the first fingerprint is the same and the second fingerprint is different, the update type flag is set to indicate semantic change; When the comparison result shows that the first fingerprint is different and the second fingerprint is different, the update type flag is set to indicate a mixed change.

5. The method according to claim 4, characterized in that, The step of invoking the corresponding processing strategy to synchronize the metadata object based on the change type indicated by the update type flag specifically includes: When the update type flag indicates a structural change or a mixed change, the technical attributes of the metadata object are updated to the technical attributes in the current status information through the structural update channel. When the update type flag indicates a semantic change or a hybrid change, the business attributes of the metadata object in the baseline state information are evaluated through the semantic decision channel, and a decision is made on whether to update the business attributes to the business attributes in the current state information based on the evaluation results.

6. The method according to claim 5, characterized in that, The step of evaluating the value of the business attributes of the metadata object in the baseline state information and deciding whether to update the business attributes to the business attributes in the current state information based on the evaluation results specifically includes: If the business attributes in the baseline status information are evaluated as high value, then the business attributes in the baseline status information are retained, and the business attributes in the current status information are ignored. If the business attribute in the baseline status information is evaluated as low value, then the business attribute is updated to the business attribute in the current status information.

7. The method according to claim 1, characterized in that, The change tracking record includes at least the following information: the unique identifier of the metadata object, the synchronization batch number, the change type, the change content summary, the merge decision, the snapshot of the key status before the change, the snapshot of the key status after the change, and the synchronization timestamp.

8. A metadata incremental synchronization and change tracking system based on intelligent fingerprinting, characterized in that, include: The data acquisition module is used to obtain the current status information of metadata objects in the data source; The fingerprint generation module is used to generate a first fingerprint representing the technical form and a second fingerprint representing the business semantics of the metadata object. The comparison module is used to compare the first fingerprint and the second fingerprint corresponding to the current state information with the first fingerprint and the second fingerprint corresponding to the pre-stored baseline state information. The status management module is used to determine the change type of the metadata object based on the comparison results, and drive the three-state flags used to indicate the object status to be updated. The three-state flags include at least an addition flag, a deletion flag, and an update type flag. The synchronization processing module is used to invoke the corresponding processing strategy according to the change type indicated by the update type flag, so as to synchronize the metadata object and generate the corresponding change tracking record.

9. The system according to claim 8, characterized in that, The first fingerprint is a structural fingerprint, generated by extracting the technical attributes of the metadata object and performing hash calculations; the second fingerprint is a semantic fingerprint, generated by extracting the business attribute text of the metadata object and converting it into a numerical vector using a natural language processing model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 7.

Images