A big data governance system and method based on a hierarchical label system

Abstract

The application belongs to the technical field of big data management, and discloses a big data management system and method based on a layered label system, which comprises a data access preprocessing unit, a label modeling and layered configuration unit, a label analysis and data binding unit, a label-driven data management unit, and a label conflict detection and dynamic updating unit. The label-driven data management unit is triggered by the label binding result, the management rules are loaded through a dynamic rule engine, and the data is cleaned, classified, and safely protected by using a multi-dimensional verification strategy and an encryption control mechanism. The label conflict detection and dynamic updating unit monitors the label system in real time, adjusts and repairs conflicts by means of path traversal and version management technology, and ensures the stability of the label system. The two units jointly automatically process abnormal data, maintain the label system, and guarantee data quality and safety.

Description

Technical Field

[0001] This invention belongs to the field of big data governance technology, specifically a big data governance system and method based on a hierarchical labeling system. Background Technology

[0002] Against the backdrop of accelerated digital transformation, enterprises and organizations are facing increasingly severe data governance challenges.

[0003] In terms of tag system construction, existing tag systems mostly adopt a static predefined mode, which cannot flexibly respond to changes in business scenarios and cross-industry application needs. When business processes are adjusted or new business forms emerge, fixed tag levels and attributes are difficult to adapt quickly, resulting in data that cannot be accurately classified and limiting data value mining.

[0004] In the tag parsing and data binding process, there is a lack of comprehensive consideration for multi-tag adaptation scenarios. Existing technologies often select and bind tags based on only a single factor, ignoring key elements such as business sensitivity and permission compliance, resulting in insufficient accuracy in tag-data matching and affecting the accuracy and reliability of data analysis.

[0005] In the data governance process, traditional data cleaning and security management methods are relatively crude and difficult to refine for data tag attributes, failing to meet enterprises' stringent requirements for data quality and security. Furthermore, the lack of real-time monitoring and dynamic repair mechanisms during tagging system operation makes it difficult to detect and handle anomalies such as tag conflicts and redundancy in a timely manner, easily leading to data management chaos and reducing data governance efficiency. Summary of the Invention

[0006] The purpose of this invention is to provide a big data governance system and method based on a hierarchical labeling system to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a big data governance system based on a hierarchical tagging system, the system comprising: a data access preprocessing unit, a tag modeling and hierarchical configuration unit, a tag parsing and data binding unit, a tag-driven data governance unit, and a tag conflict detection and dynamic update unit;

[0008] Data access preprocessing unit: responsible for receiving multi-source heterogeneous data, covering various formats, and constructing unified conversion rules through data format standardization mapping technology to convert the raw data into a standard format, which facilitates the work of subsequent label modeling and hierarchical configuration units;

[0009] Tag modeling and hierarchical configuration unit: receives preprocessed data, based on data types and business scenarios, uses tag hierarchical modeling formula to construct tag system. By setting multiple levels of tags, the relationship and attributes of the tags are clear, supporting recursive expansion and cross-industry adaptation;

[0010] Tag analysis and data binding unit: based on the system constructed by the tag modeling and hierarchical configuration unit, use multi-mode matching algorithm to analyze data, automatically bind data fields and tags according to the mapping path. Use priority formula to optimize binding selection, then output the binding result to the tag-driven data governance unit;

[0011] Tag-driven data governance unit: based on the results of the tag analysis and data binding unit, dynamically loads governance rules, automatically cleanses abnormal, redundant and other problem data. This tag-based governance mode ensures data quality, and the processed data enters the tag conflict detection and dynamic update unit for monitoring;

[0012] Tag conflict detection and dynamic update unit: real-time monitoring of data tags processed by the tag-driven data governance unit, based on hash comparison and path traversal algorithm, using detection formula to find conflicts and adjust in time to ensure the stability of the tag system.

[0013] Tag binding priority formula

[0014]

[0015] Where: P is the priority score of tag binding The weights of business tags, data type tags, and permission tags respectively; x, y, z are the number of tags hit by the current data;

[0016] Tag conflict detection formula:

[0017] D=

[0018] Where, D is the tag conflict detection result value; Hi is the current mapping hash value of the tag; Hh is the historical mapping hash value of the tag; n is the number of tags;

[0019] Tag-driven data governance formula: TiGi

[0020] Where: R is the result set after cleaning; S is the input data set; T is the first matched abnormal tag; Gi is the abnormal cleaning rule; n is the number of abnormal tags.

[0021] Preferably, the data access preprocessing unit comprises:

[0022] (1) Multi-source heterogeneous data receiving: The data access preprocessing unit undertakes the key task of receiving structured data, semi-structured data and unstructured data and other multi-source heterogeneous data. These data sources are extensive, with various formats, covering common data formats such as JSON, CSV, XML, and non-traditional data forms such as pictures and text. The diversity and complexity of the data highlight the importance of the work of this unit;

[0023] (2) Data format standardization processing: To ensure smooth progress of subsequent label parsing process, this unit adopts data format standardization mapping technology to deeply process the original data. By constructing unified data conversion rules, different formats of data are accurately converted into standard data format supporting label parsing. This process eliminates the parsing obstacles caused by data format differences, ensures that the data entering the system is consistent in structure, and effectively improves the data processing efficiency and accuracy.

[0024] Data format standardization mapping expression:

[0025]

[0026] In the formula: The standardized data set; The original data set accessed; The preset data standardization parameters (such as encoding format, data type mapping table); Data preprocessing mapping function.

[0027] In existing big data governance, the data sources are very different. If not preprocessed, the subsequent label parsing and data governance rules will be invalid. This formula effectively improves the data adaptability and provides structural compatibility guarantee for subsequent label parsing through the data standardization mapping function.

[0028] Preferably, the label modeling and hierarchical configuration unit comprises:

[0029] (1) Multi-level label system construction: After receiving the standardized data processed by the data access preprocessing unit, according to the data type and business scenario, the core work is carried out with the help of the label hierarchical modeling formula. By setting different levels of labels such as industry, business and permission, a multi-level label system is constructed to clearly define the parent-child relationship, level and specific attributes of each label;

[0030] Label hierarchical modeling expression:

[0031]

[0032] In the formula: The first The first Label of the layer; Label parent node name; a set of label attributes (such as industry attributes, permission attributes);

[0033] The conventional label management mostly adopts flat labels or static labels, which cannot adapt to complex business structures. The formula solves the problems of coarse label granularity and insufficient adaptability in the prior art by introducing label levels and attribute construction paths, and realizes dynamic label system construction.

[0034] (2) Flexible adaptation and management support: the label modeling and hierarchical configuration unit has strong flexibility, supports recursive expansion and cross-industry adaptive configuration of labels, and can flexibly adjust the label system to meet the data labeling needs in different industries and business scenarios. This feature not only meets the diversified needs, but also provides strong support for data refinement management, ensuring that data can be accurately and effectively classified and applied in different scenarios.

[0035] Preferably, the label analysis and data binding unit comprises:

[0036] (1) Data analysis and binding based on the label system: the label analysis and data binding unit closely cooperates with the label modeling and hierarchical configuration unit, and performs accurate analysis on data based on the constructed label system.

[0037] This unit uses a multi-modal matching algorithm to quickly and efficiently extract data features, and then automatically binds data fields and hierarchical labels according to the label mapping path. In this way, data has accurate label identification;

[0038] (2) Priority optimization selection of multi-label adaptation: when there are multiple adaptive label candidates, the label analysis and data binding unit uses the label binding priority formula to comprehensively consider key factors such as the sensitivity of labels to business, the degree of data adaptation, and the rationality of permissions, and scientifically determines the label binding priority; Through this priority evaluation mechanism, the most suitable label is selected for binding, effectively improving the accuracy of label analysis, ensuring accurate matching between data and labels, and enhancing the reliability of data management.

[0039] Label binding priority expression:

[0040]

[0041] In the formula, P is the label binding priority score; is the business label weight; is the data type label weight; is the permission label weight; is the number of business labels hit by the current data; is the number of data type labels hit by the current data; The number of permission tags hit for the current data.

[0042] Preferably, the label-driven data governance unit comprises:

[0043] (1) Governance rule driving based on label binding: based on the result of label binding, dynamically load targeted label-driven data governance rules, and flexibly call the rule mechanism through the label binding situation, so that the data governance work is more accurate and adaptive, and ensures that the subsequent data governance operation can be orderly and scientifically promoted;

[0044] (2) Efficient data cleaning and quality and safety guarantee: in the data governance process, through the dynamic loading of label-driven data governance rules, automatically perform deep screening on the data, accurately identify and exclude abnormal data, redundant data and format error data, and realize efficient data cleaning; This unit deeply integrates data quality control and safety management into the label system, and significantly improves the efficiency and effect of data governance through the label-driven governance mode, and comprehensively guarantees the quality and safety of data.

[0045] Label-driven data governance principle expression:

[0046]

[0047] In the formula: Result set after data cleaning Input standardized data set; The first The number of matched data anomaly labels (such as redundancy, format error); The data anomaly cleaning rule corresponding to the label The number of anomaly labels.

[0048] The traditional data governance relies on static rules and lacks dynamic label-driven paths. This formula automatically cleanses abnormal data based on label indexing, solving the problem of static rule update lag and poor data cleaning effect in the prior art.

[0049] Preferably, the label conflict detection and dynamic update unit comprises:

[0050] (1) Label conflict real-time monitoring mechanism: the label conflict detection and dynamic update unit closely connects with the label-driven data governance unit and serves as the "guardian" of the data governance process. It relies on label hash comparison and label path traversal algorithm and uses label conflict detection formula to monitor the current label mapping hash value and historical label hash value in real time. Through this continuous difference comparison, potential problems in the label system can be captured, so that label conflicts can be handled in a timely manner;

[0051] ​Tag conflict detection expression:

[0052]

[0053] In the formula: Tag conflict detection result value; The current mapping hash value of the tag The historical mapping hash value of the tag : The number of tags

[0054] The prior art relies on manual tag conflict auditing, which is low in efficiency and easy to miss. The formula quickly detects tag path changes through hash comparison, realizes real-time detection and automatic adjustment of tag conflicts, and avoids phenomena such as tag redundancy and tag failure.

[0055] (2) Dynamic adjustment to ensure system stability: Once tag conflicts or abnormal mismatching are detected, the tag conflict detection and dynamic updating unit will quickly start the response mechanism. By automatically adjusting the tag mapping path and the tag version, the contradictions in the tag system are accurately repaired to ensure the consistency and stability of the entire tag system. This dynamic updating mechanism effectively avoids the situation of data management confusion caused by tag conflicts.

[0056] The application also provides a big data management method based on a hierarchical tag system, which adopts the above-mentioned big data management system based on a hierarchical tag system. The specific steps of the method are as follows:

[0057] S1, tag modeling and hierarchical configuration: The method uses the tag modeling and hierarchical configuration unit to analyze the accessed data from multiple dimensions such as industry, business, and permissions, and establishes a multi-level tag system. In the construction process, the tag attributes are configured in detail, and the tag level relationship and the tag mapping path are clarified. In this way, a clear and orderly structural basis is provided for subsequent tag analysis and data management work, so that data can be accurately classified and managed according to different dimensions and needs;

[0058] S2, tag analysis and driving management: Based on the tag analysis and data binding unit and the tag-driven data management unit, first, the tag analysis and data binding unit is used to analyze the data, and the data fields and tags are automatically bound according to the hierarchical tag system; then, according to the data management rules driven by the tag, a series of management operations such as data classification, cleaning, desensitization, and deduplication are dynamically executed on the data layer. Through this tag-driven management mode, the high matching of data quality and tags is ensured, and the usability and security of data are effectively improved;

[0059] ​​S3, label conflict detection and dynamic update: through the label conflict detection and dynamic update unit, the conflicts, redundancies and failures that may occur in the label mapping process are monitored in real time. Using label path traversal algorithm and label version management mechanism, once an abnormality is found, the label mapping path can be quickly and dynamically updated to maintain the continuity and correctness of the label system.

[0060] Preferably, the specific steps of the label modeling and hierarchical configuration in step S1 are as follows:

[0061] S11, multi-dimensional label system construction: the label modeling and hierarchical configuration method relies on the label modeling and hierarchical configuration unit, uses the label hierarchical modeling formula, and deeply analyzes the access data from the dimensions of industry, business, and permission. By setting multiple levels of labels, a complete label system framework is constructed, facilitating fine-grained management of data;

[0062] S12, label relationship and path determination: when constructing the label system, the label attributes are configured in detail, the label hierarchical relationship is determined through the hierarchical division algorithm, and the label mapping path is determined with the help of path mapping technology, which establishes a clear and orderly structure for subsequent label analysis and data governance, and realizes precise classification management of data.

[0063] Preferably, the specific steps of the label analysis and driving governance in step S2 are as follows:

[0064] S21, label analysis and binding: based on the label analysis and data binding unit, through the multi-mode matching algorithm, the data characteristics are deeply mined to accurately analyze the data. Combined with the label binding priority formula, the business sensitivity, adaptability and permission factors are comprehensively considered, and based on the hierarchical label system and the label mapping path, the precise binding of data fields and labels is automatically realized;

[0065] S22, label-driven data dynamic governance: dynamically load label-driven data governance rules, use automatic data processing technology to perform data classification, cleaning, desensitization, and deduplication operations, and through the label-driven mode, realize the precise matching of data quality and labels, thereby improving data usability and security.

[0066] Preferably, the specific steps of the label conflict detection and dynamic update in step S3 are as follows:

[0067] S31, real-time monitoring of label abnormalities: the label conflict detection and dynamic update method relies on the label conflict detection and dynamic update unit, uses the label hash comparison and label path traversal algorithm, and combines the label conflict detection formula to scan the label mapping process in real time. By comparing the current and historical label hash values, the abnormal conditions such as conflicts, redundancies and failures are accurately located;

[0068] S32, dynamic updating: once the anomaly is monitored, the label path traversal algorithm and the label version management mechanism are used to automatically adjust the label mapping path and update the label version, so as to ensure the stability of the label system.

[0069] The beneficial effects of the present application are as follows:

[0070] 1、The present application constructs the collection architecture and the intelligent conversion engine through the data access preprocessing unit, adopts interface adaptation, syntax tree analysis, feature extraction and other technologies for multi-source heterogeneous data, and combines regular expressions and machine learning algorithms to complete format standardization, thereby reducing data processing obstacles;The label analysis and data binding unit uses a multimodal matching algorithm and a priority formula to extract features from structured, semi-structured and unstructured data using corresponding strategies, and determines the binding priority by comprehensively considering multiple factors;Through the synergistic effect of the two, the data format and label adaptation problem is avoided, and the data processing efficiency and accuracy are greatly improved.

[0071] 2、The present application constructs a multi-level label system by means of domain knowledge graph, semantic analysis algorithm, and label hierarchical modeling formula, supports label recursive expansion, and realizes real-time monitoring of the label system through the label conflict detection and dynamic updating unit, identifies abnormalities by using hash comparison and path traversal algorithm, and dynamically repairs by using version management and path adjustment technology;Under the cooperation of the two, the system can flexibly respond to industry and business changes, thereby reducing data governance costs.

[0072] 3、The present application loads governance rules through a dynamic rule engine, uses multi-dimensional verification strategies and encryption control mechanisms to realize data cleaning, classification and security protection, and realizes real-time monitoring of the label system through the label conflict detection and dynamic updating unit, adjusts and repairs after discovering conflicts by means of path traversal and version management technology, and ensures the stability of the label system;Through the joint action of the two, abnormal data is automatically processed, the label system is maintained, and the data quality and security are improved. BRIEF DESCRIPTION OF DRAWINGS

[0073] Fig. 1 The present application is based on a layered label system big data governance system flowchart;

[0074] Fig. 2 The present application is based on a layered label system big data governance method flowchart. DETAILED DESCRIPTION

[0075] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0076] As shown in the figure, the embodiment of the present application provides a big data governance system based on a hierarchical label system, which comprises a data access preprocessing unit, a label modeling and hierarchical configuration unit, a label analysis and data binding unit, a label-driven data governance unit and a label conflict detection and dynamic updating unit. Figs. 1-2

[0077] The data access preprocessing unit: as the starting point of data processing of the system, it is responsible for receiving multi-source heterogeneous data, covering multiple formats. Through data format standardization mapping technology, unified conversion rules are constructed to convert raw data into standard format, facilitating the work of the label modeling and hierarchical configuration unit;

[0078] The label modeling and hierarchical configuration unit: based on the data type and business scenario, the label hierarchy modeling formula is used to build the label system after preprocessing data. By setting multiple levels of labels, the label relationship and attributes are clear, supporting recursive expansion and cross-industry adaptation, and the results provide a label framework for the label analysis and data binding unit;

[0079] The label analysis and data binding unit: according to the system constructed by the label modeling and hierarchical configuration unit, the multi-mode matching algorithm is used to analyze the data, and the data fields and labels are automatically bound according to the mapping path. The priority formula is used to optimize the binding selection, and the accurate binding result is the premise of the work of the label-driven data governance unit;

[0080] The label-driven data governance unit: based on the results of the label analysis and data binding unit, the governance rules are dynamically loaded, and the abnormal, redundant and other problem data are automatically cleaned. This label-based governance mode guarantees data quality, and the processed data enters the label conflict detection and dynamic updating unit for monitoring;

[0081] The label conflict detection and dynamic updating unit: real-time monitoring is performed on the data labels processed by the label-driven data governance unit, based on the hash comparison and path traversal algorithm, the detection formula is used to find conflicts and adjust in time, ensuring the stability of the label system.

[0082] ​Application scenarios of multi-mode matching algorithm: structured data uses regular expressions to extract key fields, semi-structured data uses JSONPath or XPath path language to parse node relationships, and unstructured data uses NLP models such as BERT and GPT to extract semantic vectors.

[0083] The label-driven data governance unit automatically loads the corresponding data governance rules bound to the label through the label-rule mapping index library, and dynamically triggers data cleaning, classification, desensitization, and security control operations.

[0084] Label conflict detection compares the current label with historical labels in real time through label hash values, quickly traverses the label path based on depth-first search (DFS) or breadth-first search (BFS) algorithms in the graph database, locates abnormal label paths in a timely manner, and initiates a label version management mechanism to dynamically repair conflicts.

[0085] Among them, the data access preprocessing unit realizes the efficient reception and processing of multi-source heterogeneous data such as structured, semi-structured, and unstructured data by constructing a high-compatibility and intelligent data processing architecture. For structured data, standard interfaces such as JDBC and ODBC are used to seamlessly connect with relational databases, supporting real-time incremental extraction and full-load; for semi-structured data such as JSON and XML, syntax tree-based parsing technology is used to accurately identify key-value pairs and hierarchical relationships in data, combined with a streaming computing framework, it can efficiently process large amounts of log data. When facing non-structured data such as images and text, image feature extraction is performed with the OpenCV library, and NLP technology is used to convert text into word vectors or semantic units.

[0086] During the data format standardization process, the unit has an industry standard data template library built-in, supports users to customize conversion rules through a visual interface, and uses regular expressions and XSLT style sheets to realize format conversion; at the same time, machine learning algorithms are introduced, through training on historical data, automatically identify data encoding, separator features, generate the optimal conversion scheme, and use hash checksum and pattern matching dual verification mechanisms to ensure the integrity and consistency of data during the conversion process, finally output unified format data that meets the label parsing requirements, facilitating subsequent label modeling, parsing and data governance work.

[0087] Among them, the label modeling and hierarchical configuration unit immediately receives standardized data output by the data access preprocessing unit, uses domain knowledge graph construction technology and semantic analysis algorithms, combines data type characteristics and business scenario requirements, and constructs a multi-level label system through label hierarchical modeling formulas.

[0088] In the system construction process, a tree hierarchy model is adopted to subdivide business process labels and permission control labels, attribute description language is used to accurately define the value range, data type and correlation rules of each label, and a topological sorting algorithm is used to clarify the parent-child hierarchy relationship and dependency path between labels.

[0089] Meanwhile, based on the metadata-driven architecture design, the recursive expansion of labels is supported, and when new data objects or business requirements appear, sub-labels can be added in the existing hierarchy structure through a dynamic node insertion algorithm; through an adaptive parameter adjustment mechanism, the hierarchy depth, attribute configuration and mapping rules of the label system are automatically optimized according to the data characteristics, business logic and regulatory requirements of different industries, realizing adaptive configuration across industries, and facilitating subsequent label analysis, data binding and fine-grained management.

[0090] Among them, the label analysis and data binding unit realizes the deep mining of data characteristics and accurate label binding through the fusion of multi-modal matching algorithm and intelligent mapping mechanism. In the data feature extraction link, this unit adopts a hybrid multi-modal matching strategy including regular expression matching, semantic similarity calculation and machine learning model prediction: for structured data, regular expressions are used to quickly locate key data fields; for semi-structured data, JSONPath, XPath and other path languages are used to analyze node relationships, and semantic similarity algorithms are used to identify data semantics; for unstructured text data, pre-trained BERT, GPT and other natural language processing models are used to extract semantic vector features.

[0091] In the automatic binding process, the system uses the label topology structure stored in the graph database to efficiently match data fields and label nodes according to the hierarchy relationship and attribute constraints defined by the label mapping path, and introduces data lineage tracking technology to record the binding process, ensuring that each data label association has a traceable record. When multiple label adaptation scenarios occur, the label analysis and data binding unit evaluates the label business weight, data fit score and permission compliance verification results based on the label binding priority formula, and uses a multi-objective decision algorithm to sort the candidate labels, prioritizing labels that meet business requirements, have high data adaptation and meet permission requirements for binding.

[0092] Among them, the label-driven data governance unit has a built-in dynamic rule engine that pre-associates different types of labels with corresponding governance rules by establishing a label-rule mapping index library. When receiving label binding data, it can achieve millisecond-level response and load targeted data governance rule sets based on rule priority algorithms and event triggering mechanisms.

[0093] In the data cleaning link, a multi-dimensional data checking strategy is adopted: for structured data, compliance checking is performed through pre-set field format regular expressions and value range constraints; for semi-structured data, structure verification is performed by using JSONSchema or XMLDTD; and for unstructured data, an abnormal text detection model in natural language processing is used to identify sensitive information leakage and semantic contradiction. At the same time, in combination with data bloodline analysis technology, the source and processing history of data are traced, and abnormal data is located by using a correlation analysis algorithm.

[0094] At the data security management level, according to the permission attribute carried by the label, a hierarchical encryption strategy is adopted, AES-256 encryption is performed on high-sensitive data, and lightweight hash digest processing is performed on ordinary data, and a permission control mechanism combining an access control list (ACL) and a mandatory access control (MAC) is used to ensure that the data governance process and the label system are deeply integrated and cooperatively promoted.

[0095] Among them, the label conflict detection and dynamic updating unit is in the aspect of real-time monitoring mechanism of label conflict, this unit adopts a double-track parallel detection strategy: on the one hand, the label hash comparison algorithm is used to perform block hash processing on the label mapping data, and by calculating the SHA-256 or MD5 hash value, the hash fingerprint of the current label mapping is compared with the hash record of the historical version, and even slight data structure changes or attribute value modifications can be accurately identified; on the other hand, by means of label path traversal algorithm, based on the label topology structure constructed by the graph database, the depth-first search (DFS) or breadth-first search (BFS) algorithm is used to systematically scan along the parent-child hierarchical relationship and associated path of the label, which can not only detect directly conflicting label nodes, but also identify indirect logical contradictions caused by hierarchical relationship changes.

[0096] When a conflict or abnormal mismatch is detected, the dynamic adjustment module is immediately started, the historical version snapshot of the conflict label is extracted for difference analysis through the label version management mechanism, the adjustment priority is calculated by using the conflict detection formula, the automatic mapping and rerouting algorithm is used to re-plan the label mapping path, and the version number, modification time and other information in the label metadata are updated, so that the entire label system maintains structural integrity and logical consistency during the dynamic adjustment process.

[0097] The embodiment of the application also provides a big data governance method based on a hierarchical label system, which adopts the big data governance system based on the hierarchical label system.

[0098] S1, Tag Modeling and Hierarchical Configuration: With the help of the tag modeling and hierarchical configuration unit, the access data is analyzed from multiple dimensions such as industry, business, and permissions, and a multi-level tag system is established. In the construction process, the tag attributes are configured in detail, and the tag level relationship and tag mapping path are clearly defined. In this way, a clear and orderly structural basis is provided for subsequent tag analysis and data governance work;

[0099] S2, Tag Analysis and Driven Governance: First, the data is analyzed using the tag analysis and data binding unit, and the data fields are automatically bound to the tags according to the hierarchical tag system. Then, according to the data governance rules driven by the tags, a series of governance operations such as data classification, cleaning, desensitization, and deduplication are dynamically executed at the data level. Through this tag-driven governance mode, the high matching of data quality and tags is ensured;

[0100] S3, Tag Conflict Detection and Dynamic Update: Through the tag conflict detection and dynamic update unit, real-time monitoring of conflicts, redundancies, and failures that may occur during tag mapping is performed. Using the tag path traversal algorithm and tag version management mechanism, once an anomaly is found, the tag mapping path can be quickly and dynamically updated.

[0101] In step S1, the tag modeling and hierarchical configuration is based on the tag modeling and hierarchical configuration unit. When building the tag system, first use the tag level modeling formula to deeply analyze the data from multiple dimensions such as industry, business, and permissions. For the industry dimension, refer to the industry standard classification and the business characteristics of the enterprise to build an industry tag tree, such as setting "credit", "insurance", "securities" and other top-level industry tags in the financial industry;

[0102] For the business dimension, combine the actual business processes of the enterprise to divide sales, production, warehousing, and other business tags, and further refine the sub-tags of each business link; for the permission dimension, set up public, internal, confidential, and other permission tags according to the data sensitivity and access control requirements. At the same time, use the domain knowledge graph technology to assist in tag level division, use semantic analysis algorithms to mine the potential associations between data, and perfect the tag system structure.

[0103] In the process of defining tag relationships and paths, the tag attributes are configured in detail using the attribute description language, including tag name, data type, value range, description information, etc., such as setting the "transaction amount" tag with numerical type, range limit, and other attributes.

[0104] Through the level division algorithm, the tags are organized in a tree structure to determine the parent-child level relationship of each tag, and the topological sorting algorithm is used to ensure the logical consistency of the level relationship. Using path mapping technology, the directed edge relationship between tag nodes is established in the graph database, and through the setting of node attributes and edge weights, the tag mapping path is accurately described, realizing the ordering and structuring of the tag system.

[0105] In step S2, label parsing and driving governance refers to the use of a hierarchical processing strategy in the label parsing and binding process. For structured data, a field feature library is established, and regular expressions and pattern recognition techniques are used to quickly locate and extract formatted fields such as dates and numbers. For semi-structured data, JSONPath and XPath syntax are used to parse nodes, and semantic similarity calculations are used to identify the business meaning of the data. For unstructured text, pre-trained NLP models such as BERT and RoBERTa are used for word vector conversion and semantic analysis to extract key information.

[0106] In the binding process, the label binding priority formula takes into account business weight (determined by business process diagrams), data adaptation (cosine similarity between data features and label attributes), and permission compliance (verified according to enterprise data access policies). A multi-objective decision-making algorithm is used to sort the candidate labels to ensure binding accuracy.

[0107] In the label-driven data dynamic governance phase, dynamic loading is achieved through a rule engine. The rule engine has an embedded governance rule library based on the event-condition-action (ECA) model, which includes data classification rules (such as dividing data sensitivity based on the label "customer level"), cleaning rules (handling logic for null values and outliers), desensitization rules (masking and generalization algorithms), and deduplication rules (duplicate data identification based on hash values).

[0108] The system uses an automated data processing pipeline to send data to each processing module in turn. During processing, data lineage tracking technology is used to record data transformation tracks, ensuring that the governance process is traceable, and ultimately achieving deep integration and collaborative governance of data and label systems.

[0109] In step S3, label conflict detection and dynamic update refers to the use of a dual-track parallel detection strategy in the label anomaly real-time monitoring process. On the one hand, the label hashing comparison algorithm is used to map data to semantic units and split them into independent blocks. The SHA-256 hash value of each block is calculated, and the sliding window technique is used to continuously compare the current block hash value with the historical version hash record, allowing for accurate capture of subtle data structure changes or attribute value modifications. On the other hand, the label path traversal algorithm is used to build a label topology network based on a graph database. The combination of depth-first search (DFS) and breadth-first search (BFS) is used to systematically scan along the parent-child hierarchical relationships and associated dependency paths of labels. This not only identifies directly conflicting label nodes but also discovers indirect logical contradictions caused by hierarchical relationship changes through path backtracking algorithms.

[0110] At the same time, combined with the label conflict detection formula, the label attribute change frequency, the number of associated nodes and other multi-dimensional factors are comprehensively considered to calculate the weight of the monitored differences, so as to accurately locate the abnormal conditions such as conflict, redundancy and failure.

[0111] Once the anomaly is monitored, the affected label association path is first reorganized through the label path traversal algorithm, the shortest path algorithm and the topological sorting algorithm in graph theory are used to generate the optimal path adjustment scheme; at the same time, relying on the label version management mechanism, the historical version snapshot of the conflict label is called to perform difference analysis, and the version merging algorithm is used to integrate the effective change content into the current version.

[0112] During the adjustment process, the transaction management mechanism is used to ensure data consistency, the modification operation of the label mapping path is atomized, and the version number, modification timestamp, operation log and other information in the label metadata are updated synchronously, finally realizing the dynamic repair and stable maintenance of the label system.

[0113] It should be noted that in this article, relationship terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0114] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A big data governance system based on a hierarchical labeling system, characterized in that: The system includes: a data access preprocessing unit, a tag modeling and hierarchical configuration unit, a tag parsing and data binding unit, a tag-driven data governance unit, and a tag conflict detection and dynamic update unit; Data access preprocessing unit: responsible for receiving multi-source heterogeneous data, and using data format standardization mapping technology to build unified conversion rules to convert the raw data into a standard format; Tag Modeling and Hierarchical Configuration Unit: After receiving the preprocessed data from the data access preprocessing unit, it constructs a tag system using the tag hierarchy modeling formula, and clarifies the tag relationships and attributes by setting multi-level tags; Tag parsing and data binding unit: Based on the system built by the tag modeling and hierarchical configuration unit, it uses a multi-model matching algorithm to parse data, automatically binds data fields and tags according to the mapping path, and optimizes binding selection using the tag binding priority formula; Tag-driven data governance unit: After receiving the results from the tag parsing and data binding unit, it dynamically loads tag-driven data governance rules and automatically cleans up abnormal and redundant problematic data; Tag conflict detection and dynamic update unit: Real-time monitoring of data tags processed by the tag-driven data governance unit; Based on hash comparison and path traversal algorithms, using tag conflict detection formula to detect conflicts and make timely adjustments. The data access preprocessing unit includes: (1) Multi-source heterogeneous data reception: Receiving multi-source heterogeneous data including structured data, semi-structured data and unstructured data, covering common data formats such as JSON, CSV and XML, as well as non-traditional data forms such as images and text; (2) Data format standardization processing: Data format standardization mapping technology is adopted to accurately convert data of different formats into standard data formats that support tag parsing by constructing unified data conversion rules; The tag modeling and hierarchical configuration unit includes: (1) Construction of multi-level tag system: After receiving the standardized data processed by the data access preprocessing unit, a multi-level tag system is constructed with the help of the tag hierarchy modeling formula, and the parent-child relationship, level and specific attributes of each tag are clarified. (2) Flexible adaptation and management support: It supports recursive expansion of tags and cross-industry adaptive configuration. It can flexibly adjust the tag system in response to the data tagging needs of different industries and business scenarios. The tag parsing and data binding unit includes: (1) Data parsing and binding: Based on the established tag system, the data is accurately parsed, and the multi-modal matching algorithm is used to extract data features quickly and efficiently. Then, according to the tag mapping path, the data fields are automatically bound to the hierarchical tags. (2) Priority optimization of multi-tag adaptation: By using the tag binding priority formula, the tag binding priority is scientifically determined by comprehensively considering key factors such as the sensitivity of the tag to the business, the degree of data adaptation and the rationality of permissions. The tag-driven data governance unit includes: (1) Tag-based governance rule-driven: Based on the results of tag binding, targeted tag-driven data governance rules are dynamically loaded; (2) Efficient data cleaning and quality and safety assurance: By dynamically loading data governance rules driven by tags, the data is automatically screened in depth, and abnormal data, redundant data and data with incorrect format are accurately identified and removed; The tag conflict detection and dynamic update unit includes: (1) Real-time tag conflict monitoring mechanism: Based on tag hash comparison and tag path traversal algorithm, and with the help of tag conflict detection formula, the current tag mapping hash value and historical tag hash value are monitored in real time, and this continuous difference comparison is carried out. (2) Dynamic adjustment to ensure system stability: When a tag conflict or abnormal mismatch is detected, the contradictions in the tag system are accurately repaired by automatically adjusting the tag mapping path and tag version.

2. A big data governance method based on a hierarchical tagging system, employing the big data governance system based on a hierarchical tagging system as described in claim 1, characterized in that: The specific steps of this method are as follows: S1. Tag Modeling and Hierarchical Configuration: Using the tag modeling and hierarchical configuration unit, establish a multi-level tag system, configure tag attributes in detail, and clarify the tag hierarchy relationship and tag mapping path; S2, Tag Parsing and Driven Governance: The tag parsing and data binding unit parses the data and automatically binds data fields and tags according to the hierarchical tag system. Then, according to the tag-driven data governance rules, it dynamically performs data classification, cleaning, desensitization, and deduplication operations. S3, Tag Conflict Detection and Dynamic Update: The tag conflict detection and dynamic update unit monitors potential conflicts, redundancies, and failures during the tag mapping process in real time. By utilizing the tag path traversal algorithm and tag version management mechanism, the tag mapping path can be updated quickly and dynamically.

3. The big data governance method based on a hierarchical tagging system according to claim 2, characterized in that: The specific steps for label modeling and hierarchical configuration in step S1 are as follows: S11. Multi-dimensional tag system construction: Tag modeling and hierarchical configuration method relies on tag modeling and hierarchical configuration unit, uses tag hierarchical modeling formula, analyzes access data from the dimensions of industry, business and permission, and builds a complete tag system framework by setting multi-level tags; S12. Clear Tag Relationships and Paths: When constructing the tag system, the hierarchical relationship of tags is clarified through a hierarchical partitioning algorithm, and the tag mapping path is determined with the help of path mapping technology.

4. The big data governance method based on a hierarchical tagging system according to claim 3, characterized in that: The specific steps of tag parsing and driver governance in step S2 are as follows: S21. Tag parsing and binding: The tag parsing and data binding method relies on the tag parsing and data binding unit and adopts a multi-modal matching algorithm to deeply mine data features; By combining the tag binding priority formula, taking into account business sensitivity, adaptability and permission factors, and based on the hierarchical tag system and tag mapping path, the system automatically achieves accurate binding between data fields and tags. S22. Tag-driven dynamic data governance: With the help of the tag-driven data governance unit, tag-driven data governance rules are dynamically loaded, and automated data processing technology is used to perform data classification, cleaning, desensitization, and deduplication operations.

5. A big data governance method based on a hierarchical tagging system according to claim 4, characterized in that: The specific steps for tag conflict detection and dynamic updating in step S3 are as follows: S31. Real-time monitoring of label anomalies: The label conflict detection and dynamic update method relies on the label conflict detection and dynamic update unit, uses label hash comparison and label path traversal algorithm, and combines label conflict detection formula to scan the label mapping process in real time. By comparing the current and historical label hash values, it can accurately locate conflict, redundancy and failure anomalies. S32. Dynamic Update: When an anomaly is detected, this method uses a tag path traversal algorithm and a tag version management mechanism to automatically adjust the tag mapping path and update the tag version.

Images