Metadata version control and automatic structure comparison method and system based on incremental snapshot and intelligent decision

Abstract

The invention discloses a metadata version control and automatic structure comparison method and system based on incremental snapshot and intelligent decision, and the method comprises the following steps: managing a metadata version through an incremental snapshot model, and enabling the incremental snapshot model to support the complete backtracking of a historical structure through storing changed incremental information and associating with a version chain; carrying out difference analysis on the metadata structure by adopting a hierarchical progressive comparison algorithm; an intelligent DDL decision engine is adopted, and an executable DDL script is generated according to the difference analysis result; in the process of generating the DDL script, metadata consanguinity information is jointly checked so as to carry out change risk assessment. Through collaborative design of all the modules, double benefits of systematic breakthrough and conventional effect enhancement are brought, and the technical effect is directly associated with collaborative design.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data governance and metadata management, and particularly relates to a metadata version control and automatic structure comparison method and system based on incremental snapshots and intelligent decision-making. BACKGROUND

[0002] With the deepening of enterprise digital transformation, data middle platform, data warehouse and big data platform are increasingly popular. As the "specification" that defines the structure, meaning and relationship of data assets, the management capability of metadata has become the core of data governance. In the field of frequent business demand changes such as finance, e-commerce and government affairs, the metadata table structure needs to be continuously iterated in version, while facing the challenges of compliance audit (such as change traceability, error rollback), cross-environment synchronization (such as consistent structure of development, test and production environments) and operation and maintenance efficiency under large-scale metadata management (such as ten thousand tables and millions of fields).

[0003] The current mainstream metadata management scheme exposes the following systematic and associated technical defects when dealing with the above challenges: Conflict between version storage efficiency and rollback reliability: existing schemes mostly use "full snapshot" mode, resulting in exponential growth of version storage volume with the number of versions, and slow query response. While some simplified schemes using incremental storage lack effective version chain association design, which easily causes version chain breakage during rollback, resulting in high rollback failure rate, and unable to balance between "lightweight storage" and "reliable rollback".

[0004] Adaptation fault of DDL generation and execution: existing DDL automatic generation schemes are mostly limited to simple mapping of "difference to script", without considering the huge differences in DDL syntax and execution mechanism (such as online DDL support) of different databases (such as MySQL, Oracle, PostgreSQL). This results in high failure rate of the same structure change when executed on different databases, and unable to effectively use the original zero-downtime execution strategy of the database, causing long business interruption.

[0005] Lack of linkage between change impact analysis and risk assessment: existing change impact analysis is mostly conducted in isolation, disconnected from the real-time changing metadata bloodline graph. If offline bloodline data is used, risk assessment will lag; if real-time linkage is performed, comparison efficiency will be significantly slowed down. This fragmentation between "accurate analysis" and "efficient execution" makes it impossible to timely and accurately identify the "deletion of a field leading to failure of downstream ETL tasks" and other chain risks.

[0006] Therefore, in the prior art, the technical scheme is mostly in the aspect of "fragmented technology stacking", and a systematic technical framework with deep collaboration between modules and data closed loop cannot be constructed, so that the collaborative technical problems caused by the characteristics of metadata itself (such as constraint dependency, local change and multi-library heterogeneity) cannot be solved. SUMMARY

[0007] The present application aims at the above-mentioned problems, and provides a metadata version control and automatic structure comparison method and system based on incremental snapshot and intelligent decision, which adds a metadata blood relationship linkage module on the basis of the original four modules of "metadata version management, structure difference analysis, DDL script generation and visual interaction", and optimizes the core modules through "collaboration and architecture" technology sinking, to form a systematic closed loop scheme of "incremental storage-intelligent decision-blood relationship linkage", so that the modules are not simply stacked, but functionally upgraded through data flow and logical collaboration.

[0008] The technical scheme of the present application is as follows: A metadata version control and automatic structure comparison method based on incremental snapshot and intelligent decision, comprising the following steps: The metadata version is managed through an incremental snapshot model, and the incremental snapshot model stores the incremental information of changes and is associated with a version chain to support complete rollback of historical structures; A hierarchical progressive comparison algorithm is used to analyze the differences in metadata structures; An intelligent DDL decision engine is used to generate executable DDL scripts according to the difference analysis results; In the process of generating DDL scripts, the metadata blood relationship information is inquired to perform change risk assessment.

[0009] By organically combining the four core processes of incremental snapshot, hierarchical comparison, intelligent decision and blood relationship linkage, the fragmented state of each link in traditional metadata management is broken, full-link automation and intelligentization from version control, difference analysis to risk assessment are realized, and the systematic bottleneck of "fragmented technology stacking" is solved. Under the premise of ensuring data reliability and operation safety, the efficiency of metadata management is significantly improved as a whole, specifically manifested as a significant reduction in version storage overhead, an acceleration of structure comparison speed, a significant improvement in DDL execution success rate and a precise prediction of change risk.

[0010] Further, the metadata version managed through the incremental snapshot model specifically comprises: A change request of a metadata table is received, and incremental information of changes is extracted, the incremental information at least including a list of changed fields and a change type; The incremental information is stored in association with the version dependency relationship in the version chain association table to form a version chain topology; When rollback to the target version is needed, a dependent path from the current version to the target version is determined based on the version chain topology, and incremental information is superimposed according to the path to reconstruct the complete structure of the target version.

[0011] Through the collaborative design of "incremental information + version chain topology", the version storage volume is reduced by up to 60% while ensuring the complete and reliable backtracking of the structure of any historical version, and the success rate of rollback is increased to 100%, solving the contradiction between "lightweight storage" and "reliable backtracking". The precise tracking based on the dependent path and the structure integrity check during the superposition process shorten the version rollback time from minutes to seconds, and completely avoid the rollback failure caused by the broken version chain or inconsistent constraints.

[0012] Further, the hierarchical progressive comparison algorithm specifically includes: Calculate the joint hash value of the new and old version metadata structure for table-level fast screening; If the hash values are different, the fields are sorted based on the constraint relationship between the metadata elements; According to the predefined attribute priority, the sorted fields are compared in multiple dimensions and the difference degree is calculated.

[0013] Through the three-layer progressive mechanism of "table-level hash screening-field dependent sorting-multi-dimensional comparison", the collaboration of fast screening and accurate comparison is realized. Especially in the no-change scenario, the single-table comparison time is reduced from 2 seconds to less than 0.1 second, with an efficiency improvement of 20 times. Through dependent sorting, false positives caused by improper field comparison order are avoided, and through multi-dimensional priority comparison, high-impact changes are focused on, with a difference report accuracy rate of 100%, providing reliable input for subsequent DDL generation and risk assessment.

[0014] Further, the intelligent DDL decision engine adopts the architecture of "general decision framework + database feature plug-in"; Generating a DDL script includes: Through the general decision framework, based on the difference analysis results and the metadata constraint relationship, the topological sorting of the DDL execution sequence is performed; Through the database feature plug-in, according to the type of the target database, the low-impact execution strategy specific to the database is adapted; The adapted execution strategy is injected into the final generated DDL script.

[0015] Through the architecture of "general framework + database plug-in", the unified management of the core logic of DDL generation (such as dependency sorting) is realized, the rule fragmentation is avoided, and the extension is strong. Through topological sorting, the DDL execution conflict caused by constraint dependency is fundamentally eliminated, and the DDL execution failure rate is reduced from 30% to 0% combined with cross-database strategy adaptation.

[0016] Further, the topological sorting of the DDL execution sequence adopts an improved Kahn algorithm containing dependency loop detection; when a dependency loop is detected, an intermediate DDL for temporarily releasing and restoring the constraint is automatically generated.

[0017] Through the above method, the "dependency loop" deadlock problem that cannot be handled by the prior art can be automatically detected and cracked, and through the generation of intermediate step scripts, full automation of DDL generation in complex constraint scenarios is realized, reducing 100% of manual intervention debugging, which is a significant technical progress.

[0018] Further, the association metadata blood relationship information specifically includes: Preferably, the blood relationship data of recent high-frequency access is queried from the cache database; If the cache is not hit, the graph database storing the complete blood relationship map is queried; Based on the query result, the risk value of the change is calculated in combination with the difference degree obtained in the difference analysis.

[0019] Through the query mechanism of "cache priority + graph database backup", the risk assessment association time is controlled within 0.5 seconds on the premise of ensuring the freshness of the blood relationship data, realizing accurate identification of chain risk while not significantly increasing the overall time consumption of the comparison process. By combining the "difference degree" at the technical level with the "dependency importance" at the business level to calculate the risk value, the risk assessment result is more objective and accurate, and the identification accuracy is improved from 50% to 100%, providing a scientific basis for change approval and execution.

[0020] The application also includes a metadata version control and automated structure comparison system based on incremental snapshots and intelligent decision-making, and executes a metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making, including: The metadata version management module manages the metadata version through an incremental snapshot model, and the incremental snapshot model stores the incremental information of the change and is associated with the version chain to support complete rollback of the historical structure; The structure difference analysis module adopts a hierarchical progressive comparison algorithm to analyze the differences in the metadata structure; The DDL script generation module adopts an intelligent DDL decision engine to generate executable DDL scripts according to the difference analysis result; The metadata blood relationship linkage module inquires metadata blood relationship information in the process of generating the DDL script to perform change risk assessment.

[0021] Further, a visual interaction module is further included, and the visual interaction module is configured to: In the same interface, structural differences are displayed in different colors associated with risk levels; In response to an interactive operation of a user on a difference field, a downstream blood relationship dependent link of the field is displayed; For changes of different risk levels, corresponding collaborative operation suggestions and process guidance are provided.

[0022] Through the above system, structural differences, risk levels, blood relationship links and operation suggestions are fused in the same interface to provide a "one-stop" change panoramic view for users, greatly reducing the understanding cost and operation threshold. According to the risk level, collaborative operation suggestions are dynamically provided, the best practice is solidified into the process, the user is guided to perform standardized and safe change operation, the risk of human error is reduced, and the overall operation efficiency and compliance are improved.

[0023] The application further includes an electronic device including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements a metadata version control and automatic structure comparison method based on incremental snapshots and intelligent decision-making when executing the program.

[0024] The application further includes a computer readable storage medium having a computer program stored thereon, and the program implements a metadata version control and automatic structure comparison method based on incremental snapshots and intelligent decision-making when executed by a processor.

[0025] Compared with the existing technology, the application has the following advantages: 1. Through the collaborative design of each module, "system breakthrough" and "conventional effect strengthening" dual benefits are brought, and the technical effect is directly related to the collaborative design; 2. Efficiency improvement: single table comparison time is reduced from 2 seconds to 0.1 seconds (in the case of no change, resulting from "table-level hash screening + difference site guidance"), version rollback time is reduced from 2 seconds to 0.5 seconds (resulting from "dependency path tracing + incremental superposition optimization"), and DDL generation + verification time is reduced from 5 seconds to 2 seconds (resulting from "universal framework automation + plug-in adaptation"); 3. Compliance enhancement: supporting incremental tracing of any version within 12 months (resulting from "version chain topology complete record dependency relationship"), meeting the "full link auditability" requirements of SOX and GDPR, and audit response time is reduced from 10 seconds to 1 second (resulting from "incremental snapshot quick query + blood relationship link real-time display"); 4. Reduced operational costs: 90% reduction in manual intervention (e.g. DDL debugging, risk investigation, resulting from "DDL zero failure + automatic risk identification"), and 40 hours of repetitive labor time reduction per month for data engineers. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 Collaborative process diagram for metadata version management.

[0027] Figure 2 Collaborative algorithm flowchart for structural difference comparison.

[0028] Figure 3 Collaborative engine workflow diagram for intelligent DDL decision.

[0029] Figure 4 Collaborative module workflow diagram for metadata bloodline linkage. DETAILED DESCRIPTION

[0030] It should be noted that the terms "first" and "second" and the like such relational terms are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0031] The existing solution does not break through the "fragmented technology stacking" limitation and does not build a systematic technical framework for metadata management: ① Version management only focuses on "storage form" and does not design a closed-loop mechanism of "incremental information-version chain-backtracking logic", resulting in incoordination between storage and backtracking; ② DDL management separates "dependency analysis, database adaptation, execution strategy", and does not form an integrated engine of "difference input-intelligent decision-adaptation output", which cannot meet the needs of multi-database differences and zero downtime; ③ Risk assessment does not establish a linkage model of "difference comparison-real-time bloodline-risk calculation", and regards bloodline data as static auxiliary information rather than the core input of dynamic risk assessment, resulting in low risk identification accuracy.

[0032] Key technical difficulties: ① "Lightweight-reliable" collaborative design of incremental snapshot: Metadata changes have "locality" (such as modifying only a single field constraint), but need to ensure that the incremental information can be traced back to the complete history structure. If only the changed field is stored, it is easy to cause rollback failure due to version chain breakage. If redundant associated information is stored, the incremental advantage is lost. A "incremental field + version chain topology" model needs to be designed to ensure the integrity of the traceability based on lightweight storage. This requires breaking through the traditional linear version chain design and introducing mesh association logic. ② "General-adaptation" balance of cross-database DDL decision: The DDL syntax (such as field type, constraint syntax) and execution mechanism (such as online redefinition, index creation method) of different databases (MySQL / Oracle / PostgreSQL) differ significantly. A "general decision framework + database feature plug-in" architecture needs to be built. The general framework uniformly processes the dependency sorting and risk assessment logic, and the plug-in layer adapts to the execution strategy specific to each database, avoiding rule fragmentation while ensuring the extensibility of the framework to support the addition of new database types. ③ "Real-time-efficient" collaborative optimization of blood relationship linkage: The metadata blood relationship graph is dynamically updated with business changes (such as adding ETL task associated fields), and real-time blood relationship data needs to be obtained to ensure accurate risk assessment during version comparison. However, real-time inquiry increases comparison time consumption. A "hotspot blood relationship cache + incremental blood relationship synchronization" mechanism needs to be designed to improve query efficiency by caching high-frequency change blood relationship data for nearly 30 days, while ensuring cache data freshness with second-level incremental synchronization, balancing real-time and performance.

[0033] The features and performance of the present application will be further described in detail below in conjunction with embodiments.

[0034] Please refer to Figures 1-4 A metadata version control and automated structure comparison method based on incremental snapshot and intelligent decision-making, comprising the following steps: The metadata version is managed by an incremental snapshot model, which stores the incremental information of changes and associates it with the version chain to support complete traceability of the history structure. A hierarchical progressive comparison algorithm is used to analyze the differences in the metadata structure. An intelligent DDL decision engine is used to generate executable DDL scripts based on the results of the difference analysis. During the generation of the DDL script, the metadata blood relationship information is inquired to assess the change risk.

[0035] As Figure 1 shown, the metadata version is managed by an incremental snapshot model, which specifically includes: Receive the change request of metadata table, call the difference analysis module to extract the change delta information (delta_field_list, delta_type), and query the version chain association table to obtain the historical version dependency relationship; If it is the first creation, store the full snapshot, and create the initial version chain in the version chain association table. If it is subsequent change, only store the incremental information, and associate the previous version as the reference (delta_base_version), while adding a "direct dependency" record in the version chain association table to build a mesh version chain topology; When rolling back, query the dependency path from the current version to the target version based on the version chain association table, superimpose the incremental snapshot on the target version based structure according to the path, and verify the integrity of the superimposed structure to achieve fast and reliable rollback.

[0036] As shown in Figure 2 , the hierarchical progressive comparison algorithm specifically includes: Calculate the joint hash value of "table_name+field_list (field name+type+primary key constraint)" of new and old versions, record the hash difference site, and quickly screen the no-structure change scenario; Based on the metadata constraint relationship and the current table primary key / foreign key definition, the fields are sorted in dependency, ensuring that the dependent fields are compared first; Combined with the hash difference site, the field dimension is compared according to the priority of "field type→constraint→length→comment", and the "dimension influence coefficient" and "field importance coefficient" are introduced to calculate the difference degree; Based on the complete structure of new and old tables, the relevance of changed fields in the difference report is verified to avoid logical errors caused by partial comparison.

[0037] The intelligent DDL decision engine adopts the architecture of "general decision framework+database feature plug-in"; As shown in Figure 3 , the generation of DDL script includes: According to the metadata difference report and the current table constraint relationship, build the change dependency graph, and use the improved Kahn algorithm with "dependency loop detection" to generate DDL execution topological sorting. If a dependency loop is detected, automatically generate intermediate release / resume constraint steps; Build the "general decision framework+database feature plug-in" architecture, match the basic strategy from the general strategy warehouse, load the corresponding database feature plug-in, and adapt the specific execution strategy (such as MySQL's INPLACE algorithm, Oracle's online redefinition) combined with the table data volume; The linkage metadata bloodline library (preferential query cache) obtains downstream dependencies of the changed field, calculates a risk value in combination with the difference degree and the dependency importance, generates a risk report, and automatically triggers downstream notification and approval processes for high-risk changes, and directly provides a DDL execution entry for medium / low-risk changes.

[0038] As shown in Figure 4 , the joint investigation of metadata bloodline information specifically includes: The difference report generated by the hierarchical progressive comparison algorithm, the metadata bloodline graph (including downstream dependency links), and the risk value calculation results are associated to form an integrated "difference-bloodline-risk" data; The change type and risk level are displayed in color layers, and the color depth increases with the increase of the risk value; Click the difference field to pop up a window to display the complete link graph of "changed field -> downstream dependency", and mark the difference impact range.

[0039] For different risk levels, provide collaborative operation suggestions and step guidance for "backup-notification-approval-execution", and realize one-stop change management.

[0040] The application also includes a metadata version control and automatic structure comparison system based on incremental snapshots and intelligent decision-making, which executes a metadata version control and automatic structure comparison method based on incremental snapshots and intelligent decision-making, including: As shown in Figure 1 , the metadata version management module: for the "storage-backtracking" collaboration problem, a three-in-one incremental snapshot model of "incremental field + version chain topology + backtracking algorithm" is designed, rather than only storing incremental information, to realize the collaboration of "lightweight storage" and "reliable backtracking".

[0041] (1) Data model design: define a three-table collaborative model of "current metadata table (DataTable)" and "history version table (DataTableHis)" "version chain association table (VersionChainLink)", and the core fields are as follows: DataTable (current table): retain the original field, add delta_base_version (incremental benchmark version number, indicating which version the current structure is based on for incremental superposition generation), and version_chain_id (unique identifier associated with version chain topology); DataTableHis (history table): add three core fields: delta_field_list: JSON format, store field information (including field name, pre-change value, post-change value, and change dimension) of changes, such as: {"field_name":"user_age","old_value":{"type":"INT","length":10},"new_value":{"type":"INT","length":3},"changed_dimension":["length"]}; delta_type: delta type (field add / del / modify, constraint add / del), used for quick screening of change type; version_chain_id: associated with VersionChainLink; VersionChainLink (Version Chain Association Table): stores the version chain topology relationship, including version_chain_id (version chain unique identifier), source_version (source version number), target_version (target version number), dependency_type (dependency type, such as "direct dependency" "indirect dependency"), solves the problem of easy breakage of traditional linear version chain backtracking.

[0042] (2) Incremental snapshot generation logic: when the user submits metadata changes, the system executes the "difference extraction-version chain construction-incremental storage" collaborative process: Call the analyzeDeltaWithDependency(DataTable old, DataTable new, VersionChainLink chain) method of DeltaAnalyzer.java to compare the new and old structures and extract delta_field_list and delta_type; not only compare the new and old structures to extract delta_field_list and delta_type, but also query the historical version dependency relationship through VersionChainLink to ensure that the incremental information is compatible with the historical version chain; If it is the first creation (no historical version), generate a full snapshot (delta_field_list is the complete field list, delta_base_version=0); at the same time, create an initial version chain (source_version=0, target_version=1, dependency_type="initial") in VersionChainLink; If it is a subsequent change, only store delta_field_list and set delta_base_version = current version number - 1; at the same time, add a related record in VersionChainLink (source_version = current version number - 1, target_version = current version number, dependency_type = "direct dependency") to build a mesh version chain topology; Incremental information is written to DataTableHis, and the version number still follows the rule of "incrementing the major version number for structural changes (e.g., V3), and incrementing the minor version number for non-structural changes (e.g., V3.1)". All versions are associated with VersionChainLink via version_chain_id.

[0043] (3) Incremental rollback optimization: Using the rollbackByChain(LongtableId, Integer targetVersion) method in DataTableHisService.java, a collaborative rollback of "precise tracing - incremental overlay - structural verification" is implemented based on the version chain topology: Based on tableId and targetVersion, query the "dependency path" from the current version to the target version through VersionChainLink (instead of simple reverse tracing) to ensure that no incremental snapshots of indirect dependencies are missed; Calling DeltaMerger.java The mergeDeltaWithValidation(tableId, targetVersion, VersionChainLinkchain) method overlays incremental snapshots onto the target version's infrastructure according to the dependency path, while simultaneously validating the integrity of the overlaid structure in real time (such as whether foreign key associations are valid). Update the current structure of the DataTable, archive the current version incremental snapshot in DataTableHis (with the operation_type marked as "rollback before"), and add a "rollback dependency" record in VersionChainLink. The rollback time is reduced from 10 seconds to within 2 seconds, and the rollback success rate is increased to 100%.

[0044] like Figure 2 As shown, the structural difference analysis module: To address the hierarchical relationship of metadata "table-field-constraint", a four-layer collaborative comparison algorithm is designed, consisting of "table-level hash filtering, field dependency sorting, multi-dimensional weight comparison, and difference verification". This algorithm overcomes the limitation of the general diff algorithm's "single-dimensional comparison" and achieves a synergistic improvement in efficiency and accuracy.

[0045] (1) Comparison logic: First layer: Table-level hash quick screening: Calculate the joint hash value of "table_name + field_list (field name + type + primary key constraint)" of the new and old versions (compared to the original scheme, increase the primary key constraint dimension, reduce false positives). If the hash value is the same, it is directly determined that there is no structural change, and the subsequent comparison is skipped. The "no change scenario" of single table comparison time is reduced from 2 seconds to 0.1 seconds; if the hash value is different, carry the hash difference site (such as "field_list type mismatch") to the next step, and provide direction for field-level comparison; Second layer: Field-level dependency sorting: Call the sortByDependencyWithConstraint(List <field>The method of sorting based on metadata constraint relationships (such as foreign key depending on primary key, index association field) and combining the primary key / foreign key definition of the current table ensures that the dependent field (such as the primary key field) is prioritized, avoiding "false modification" caused by chaotic field order (such as association relationship misjudgment caused by comparing foreign key fields first). Third layer: multi-dimensional priority comparison: Define the priority of field comparison dimensions: field type (weight 40%) > constraint (weight 30%) > length (weight 20%) > annotation (weight 10%), through the compareWithPriority(Fieldold,Field new,String hashDiff) method of DimensionComparator.java, combine table-level hash difference sites (such as "type mismatch" to prioritize type dimension verification), and compare according to priority: If the high-priority dimension (such as type) is different, it is directly marked as "modified" and there is no need to compare low-priority dimensions. If only low-priority dimensions (such as annotations) are different, mark them as "minor modifications" and classify them separately in the difference report.

[0046] Fourth layer: difference degree verification: Call the validateDiff(DiffReport report, DataTableold, DataTable new) method of DiffValidator.java based on the complete structure of the new and old tables to verify the association of "modified / deleted" fields in the difference report (such as whether there is an associated primary key before deleting the foreign key field), avoiding logical errors caused by partial comparison, and improving the accuracy of the difference report to 100%.

[0047] (2) Difference degree calculation optimization: Table-level difference degree: , Where "dimension influence coefficient" is set according to the influence degree of the dimension on the table structure (such as primary key type change coefficient is 1.2, annotation change coefficient is 0.5). For example: table name unchanged, primary key annotation modified (weight 10%, coefficient 0.5), then More accurate reflection of the actual impact of changes; Field-level difference degree: , Where "field importance coefficient" is set according to whether the field is a primary key / foreign key (primary key coefficient 1.5, normal field coefficient 1.0). Example: 2 field changes (field 1 is a primary key type change, weight 40%, coefficient 1.5; field 2 is a normal field comment change, weight 10%, coefficient 1.0), then , avoiding "one-size-fits-all" bias.

[0048] As shown in Figure 3 , the DDL script generation module: On the basis of original template rendering, an intelligent DDL decision engine of "general decision framework + database feature plug-in" collaborative architecture is constructed to realize the whole-process collaboration of "difference analysis - dependency sorting - cross-database adaptation - risk assessment - script generation", rather than simply superimposing each functional step.

[0049] (1) DDL execution order topological sorting: Call the buildGraphWithConstraint(DiffReport report, DataTable table) method of DependencyGraph.java to construct a "change dependency graph" (node = change operation, edge = constraint dependency relationship, such as "modify field F1" depends on "delete foreign key FK_F1", "add index IX_F1" depends on "modify field F1") based on the difference report and the current table constraint relationship (rather than only the difference report); Topological sorting of the dependency graph is performed using an improved Kahn algorithm (with new "dependency loop detection" logic). If a dependency loop is detected (such as "modify F1 depends on delete FK_F1, delete FK_F1 depends on modify F1"), an intermediate step of "temporarily remove constraints -> execute changes -> restore constraints" is automatically generated to avoid sorting failure; A conflict-free execution order is generated (such as "delete FK_F1 -> modify F1 -> add IX_F1"), and the DDL execution failure rate is reduced from 30% to 0%.

[0050] (2) Cross-database differentiated execution strategy: A collaborative architecture of "DBPolicyRepository (policy repository) + DBFeaturePlugin (database feature plug-in)" is constructed: DBPolicyRepository: stores general execution strategy templates (such as "field length modification needs to prioritize data range verification"); DBFeaturePlugin: Develop plugins by database type (MySQL / Oracle / PostgreSQL), encapsulate the execution logic of each database (such as the ALGORITHM parameter processing in the MySQL plugin, and the DBMS_REDEFINITION call logic in the Oracle plugin); When generating DDL, the engine performs the "general strategy matching → plugin feature adaptation → script injection" collaborative process: Call the matchPolicy(DBType type, ChangeScene scene) method of DBPolicyMatcher.java to match the general strategy from DBPolicyRepository; Load the DBFeaturePlugin corresponding to the database, and call the adaptPolicy(Policy policy, DataTable table) method to adapt the specific strategy combined with the current table data volume (such as automatically adjusting the Oracle online redefinition parameters when the data volume exceeds 10 million); Inject the adapted strategy into the DDL script, for example: Table 1 shows the DDL script after injection

[0051] (3) Bloodline linkage risk assessment: Embed risk assessment into the DDL generation process to achieve "difference → bloodline → risk → script control" collaboration: After generating the difference report, call the queryImpactWithCache(DiffReport report) method of BloodlineLinker.java to query the downstream dependencies of the changed field from the Redis cache of the bloodline module first, and if the cache is not hit, query the Neo4j graph database in real time to ensure that the query time is greater than 0.5 seconds; Call the calculateRisk(ImpactResult result, DiffReport report) method of RiskCalculator.java to calculate the risk value based on the difference degree (such as a field-level difference degree of 35%) and the importance of downstream dependencies (such as a core ETL task weight of 1.0 and a non-core report weight of 0.3): , High risk: (such as deleting a field associated with 3 core ETL tasks, with a difference degree of 100%, ); Medium risk: ; Low risk: ; Risk report generation: If it is a high-risk change, automatically trigger "downstream notification" (send an email to the ETL responsible person with change details and dependency link diagram), and prompt "need to be approved before execution" in the DDL generation interface popup; if it is a medium / low risk, directly provide DDL script execution entry, the risk assessment accuracy rate is improved from 50% to 100%.

[0052] As shown in Figure 4 , the metadata blood relationship linkage module: As an independent module and a deep collaboration with other modules, it builds a "blood construction-storage-caching-linkage-risk analysis" collaborative system, rather than just maintaining the blood graph, and realizes real-time data flow with the version management and difference analysis modules.

[0053] (1) Blood graph construction: Call the parseMultiSource(String sourceType, String content) method of BloodlineParser.java to support multi-source script parsing: ETL script (Flink SQL, DataX configuration): extract the "source field -> ETL task -> target field" association; Report SQL: extract "report -> query field -> source table field" association; After parsing, generate standardized blood data (including field ID, task ID, report ID, association type) to avoid fragmentation of different source script parsing results.

[0054] (2) Blood data storage and caching: Use "Neo4j graph database + Redis cache" collaborative storage: Neo4j graph database: store complete blood graph, establish edge connection of "field node-task node-report node", and support complex link query (such as "field A -> task 1 -> field B -> report 2"); Redis cache: store the association blood data of the changed field in the last 30 days (such as the tasks and reports associated with field A in the last 30 days) according to the "field ID + latest update time" dimension, with an update frequency of more than 1 minute (through the incremental synchronization mechanism, only synchronize the newly added / modified blood relationship); (3) Real-time linkage mechanism: Establish a "trigger-response-feedback" collaborative process with the version management and difference analysis modules: Trigger: version comparison / DDL generation module initiates blood query request (carrying field ID, query type such as "risk assessment"); Response: The blood module queries the Redis cache first, and if it hits, it returns directly; if not, it queries Neo4j and synchronizes the results to Redis. Feedback: Return "blood link data + risk level suggestion", synchronize to difference report / DDL interface, ensure that the linkage time is less than 0.5 seconds.

[0055] Visual interaction module: On the basis of the original highlight display, a "difference display-blood link-risk prompt-operation suggestion" four-in-one collaborative interface is built, rather than simply superimposing the risk dimension, realizing the user's "one-stop understanding of the whole change".

[0056] Color layering and risk association: green (new, low risk), yellow (modified, medium risk), red (deleted, high risk), orange (slight modification, no risk), color depth increases with risk value (such as high-risk red, "light red-deep red", for dark red); Blood pop-up and difference linkage: click on the difference field, the pop-up window displays the complete link diagram of "changed field→downstream dependency" (node color consistent with risk level, core task / report marked red), and marks "difference impact range" (such as "the field modification affects 2 ETL tasks, of which 1 is a core task"); Operation suggestion and process guidance: provide collaborative operation suggestions for different risk levels: High risk: automatically generate "backup data (provide backup script entry)→notify downstream (one-click send email)→submit approval (jump to approval process)→execute DDL" step guidance; Medium risk: suggest "execute during off-peak hours, and verify downstream task status before execution"; Low risk: directly provide "one-click execute DDL" entry.

[0057] On the basis of the original "metadata version management, structure difference analysis, DDL script generation, visual interaction" four modules, a metadata blood linkage module is added, and the core module is optimized through "collaboration and architecture" technology sinking, forming a "incremental storage-smart decision-making-blood linkage" systematic closed-loop solution. Each module is not simply superimposed, but through data flow and logical collaboration to realize functional leap. Through the above methods and systems, the core improvement effect of breaking through "conventional technical effect" can be achieved, as follows: Table 2 Core Improvement Effect Table

[0058] The above embodiments only express the specific implementation of the present application, which is described in more detail and specifically, but cannot be understood as a limitation to the protection scope of the present application. It should be noted that for those skilled in the art, without departing from the technical concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.< / field>

Claims

1. A method for metadata version control and automated structure comparison based on incremental snapshots and intelligent decision-making, characterized in that, Includes the following steps: Metadata versions are managed through an incremental snapshot model, which stores incremental information about changes and associates it with a version chain to support a complete backtracking of the historical structure. A hierarchical, progressive comparison algorithm is used to perform difference analysis on the metadata structure; An intelligent DDL decision engine is used to generate executable DDL scripts based on the difference analysis results; During the generation of DDL scripts, metadata lineage information is retrieved to assess the risk of changes.

2. The metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making according to claim 1, characterized in that, The management of metadata versions through the incremental snapshot model specifically includes: Receive change requests from the metadata table, extract incremental information of the changes, and the incremental information shall include at least the list of changed fields and the change type; Incremental information is associated with version dependencies in the version chain association table and stored to form a version chain topology; When it is necessary to roll back to the target version, the dependency path from the current version to the target version is determined based on the version chain topology, and incremental information is superimposed on the path to reconstruct the complete structure of the target version.

3. The metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making according to claim 1, characterized in that, The hierarchical progressive comparison algorithm specifically includes: Calculate the combined hash value of the old and new version metadata structures to perform fast table-level filtering; If the hash values ​​are different, the fields are sorted according to their dependencies based on the constraints between metadata elements; Based on predefined attribute priorities, the sorted fields are compared in multiple dimensions, and the degree of difference is calculated.

4. The metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making according to claim 1, characterized in that, The intelligent DDL decision engine adopts an architecture of "general decision framework + database feature plugin"; Generating DDL scripts includes: Using a general decision-making framework, based on the results of difference analysis and metadata constraints, the topological ordering of the DDL execution sequence is performed; By using database feature plugins, we can adapt the low-impact execution strategies specific to the target database type. The adapted execution strategy is then injected into the final generated DDL script.

5. The metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making according to claim 4, characterized in that, The improved Kahn algorithm, which includes dependency cycle detection, is used when performing topological sorting of the DDL execution sequence; when a dependency cycle is detected, an intermediate DDL is automatically generated to temporarily release and restore constraints.

6. The metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making according to claim 1, characterized in that, The linked metadata lineage information specifically includes: Prioritize retrieving frequently accessed lineage data from the cached database; If the cache is not hit, then query the graph database that stores the complete kinship map; Based on the query results and the degree of difference obtained from the difference analysis, the risk value of the change is calculated.

7. A metadata version control and automated structure comparison system based on incremental snapshots and intelligent decision-making, characterized in that, Performing the metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making as described in any one of claims 1-6 includes: The metadata version management module manages metadata versions through an incremental snapshot model. The incremental snapshot model stores incremental information about changes and associates it with the version chain to support complete backtracking of the historical structure. The structural difference analysis module uses a hierarchical progressive comparison algorithm to perform difference analysis on the metadata structure. The DDL script generation module uses an intelligent DDL decision engine to generate executable DDL scripts based on the difference analysis results. The metadata lineage linkage module queries metadata lineage information during the generation of DDL scripts to assess the risk of changes.

8. The metadata version control and automated structure comparison system based on incremental snapshots and intelligent decision-making according to claim 7, characterized in that, It also includes a visual interaction module, which is configured as follows: Within the same interface, structural differences are displayed using different colors associated with risk levels; In response to user interaction with a difference field, display the downstream lineage dependency chain of that field; For changes in different risk levels, corresponding collaborative operation suggestions and process guidance are provided.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a metadata version control and automated structure comparison method based on incremental snapshots and intelligent decision-making as described in any one of claims 1-6.

Images