Data lake treatment method and device, electronic equipment and program product

Abstract

The invention discloses a data lake treatment method and device, electronic equipment and a program product, and relates to the technical field of big data. The method comprises the steps of obtaining original data from a heterogeneous data source, constructing a formal context and generating a concept lattice; semantic embedding and similarity calculation are carried out on the field names, clustering is carried out in combination with the concept lattice, unified field names are generated, and a unified data view is constructed; performing semantic enhanced metadata management on the unified data view, including constructing an attribute graph, mapping into a knowledge graph and expanding an implicit relationship thereof, and constructing an index to support semantic query; and dynamic access control and metadata integrity management are carried out on the unified data view, so that the safety and reliability of the data are ensured. According to the method, the automation degree and accuracy of data integration are effectively improved, and the discoverability, interoperability and safety of the data are enhanced.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data lake governance method, apparatus, electronic device, and program product. Background Technology

[0002] In the era of big data, data lakes, with their centralized storage and processing capabilities for heterogeneous data, have become the core infrastructure for enterprise data asset management and value mining. As a key governance component of a data lake system, unified management and semantic enhancement of metadata directly affect data discoverability, interoperability, and analytical value, and are crucial guarantees for achieving the FAIR (Find, Access, Interoperability, Reusability) principle. Existing data lake governance solutions generally have significant limitations: FCA (Fixed-Field Analysis) relies on manual operations and is difficult to scale to large-scale scenarios; the lack of semantic association in metadata leads to difficulties in cross-domain data discovery; and static permission control mechanisms cannot adapt to dynamically changing heterogeneous data environments. Especially when field naming is inconsistent and a unified semantic standard is lacking, it is difficult to guarantee the accuracy and automation of data integration, severely restricting the application effectiveness of data lakes in advanced analytics, machine learning, and cross-organizational collaboration.

[0003] Therefore, there is an urgent need to provide a data lake governance method that can achieve automated heterogeneous data fusion, semantic enhancement of metadata, and adaptive access control. Summary of the Invention

[0004] This application provides a data lake governance method, apparatus, electronic device, and program product.

[0005] Firstly, this application provides a data lake governance method, including: Obtain raw data from heterogeneous data sources; A formal context is constructed based on the acquired raw data; wherein, the formal context includes a set of objects, a set of attributes, and binary relations between objects and attributes; Generate concept lattices based on formal context; Semantic embedding is performed on the field names of the original data to generate semantic vectors for the field names; Calculate the semantic similarity between field names based on semantic vectors; The field names in the original data are clustered and grouped according to concept lattice and semantic similarity, and a unified field name is generated for each group. A unified data view is built based on a unified field name.

[0006] In some embodiments, partitioning and compressing the formal context storage includes: Calculate partition priority scores based on the access frequency and semantic importance of field names; Partitioning the formal context based on partition priority scoring; Compress and store the formal context after partitioning.

[0007] In some embodiments, semantically enhanced metadata management of the unified data view includes: Extract metadata information from the dataset in the unified data view. The metadata information includes at least one of the following: title, author, creation time, and storage path. Construct an attribute graph to store metadata information; where nodes represent datasets and edges represent relationships between datasets. Map attribute graphs to resource description framework triples to construct knowledge graphs; Extending implicit relationships in knowledge graphs based on RDF schema rules.

[0008] In some embodiments, a B+ tree index is constructed for the knowledge graph; In response to a query request, semantic queries are performed based on the indexed knowledge graph and the expanded implicit relationships.

[0009] In some embodiments, access control of the unified data view includes: Calculate business importance scores based on business tags and access frequencies in the dataset of the unified data view; Access control rules are dynamically generated based on business importance scores and semantic labels of the dataset. Monitor access behavior to the unified data view and detect abnormal access based on the sequence of access behavior; When abnormal access is detected, the abnormal access is logged and access permissions are adjusted according to access control rules.

[0010] In some embodiments, governance of the integrity of metadata in a unified data view includes: Weights are assigned to the rules for assessing metadata integrity; Based on the business importance score of the dataset, adjust the weight of the corresponding rules and calculate the weighted integrity score; If the weighted integrity score is lower than a preset threshold, a metadata repair task for the dataset will be triggered.

[0011] Secondly, this application provides a data lake governance device, comprising: The data acquisition module is configured to acquire raw data from heterogeneous data sources; The formal context building module is configured to build a formal context based on the acquired raw data; the formal context includes a set of objects, a set of attributes, and binary relations between objects and attributes; The concept lattice generation module is configured to generate concept lattices based on formal context. The semantic embedding module is configured to perform semantic embedding on the field names of the original data and generate semantic vectors for the field names; The similarity calculation module is configured to calculate the semantic similarity between field names based on semantic vectors; The field unification module is configured to cluster and group field names in the original data based on concept lattice and semantic similarity, and generate a unified field name for each group. The view building module is configured to build a unified data view based on a unified field name.

[0012] In some embodiments, a storage optimization module is further included, the storage optimization module comprising: The priority score calculation submodule is configured to calculate the partition priority score based on the access frequency and semantic importance of the field name; The adaptive partitioning submodule is configured to partition the formal context based on partition priority scores. The hierarchical compression submodule is configured to compress and store the partitioned formal context.

[0013] Thirdly, this application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement any one of the data lake governance methods.

[0014] Fourthly, this application provides a computer program product, including a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements any one of the data lake governance methods.

[0015] The at least one technical solution adopted in this application can achieve the following beneficial effects: by constructing formal contexts, generating concept lattices, and using semantic embedding and clustering based on pre-trained models, the problem of inconsistent field naming is solved, improving the accuracy and efficiency of data integration; furthermore, by constructing attribute graphs, mapping to RDF knowledge graphs, and expanding semantic relationships based on RDFS rules, the semantic depth and machine understandability of metadata are significantly enhanced, improving data discoverability and cross-domain interoperability; furthermore, by using dynamic permission generation based on business importance scoring and semantic tags, sequence anomaly detection, and metadata integrity governance combined with dynamic weights, an adaptive data security and quality control system is constructed. The high-quality, semantically consistent, and secure data constructed in this application can effectively improve the accuracy of data analysis and machine learning, reduce the cost of cross-system data integration and governance, and ensure the security and compliance of data during flow and use.

[0016] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 This illustration schematically shows a unified data view construction process of a data lake governance method according to an embodiment of this application; Figure 2 This illustration schematically shows a formal context partitioning process diagram of a data lake governance method according to an embodiment of this application; Figure 3 This illustration schematically shows a formal context hierarchical compressed storage process of a data lake governance method according to an embodiment of this application; Figure 4 This illustration schematically shows a knowledge graph construction process of a data lake governance method according to an embodiment of this application; Figure 5 This illustration schematically shows a knowledge graph semantic enhancement process of a data lake governance method according to an embodiment of this application; Figure 6 This illustration schematically shows a knowledge graph index construction process of a data lake governance method according to an embodiment of this application; Figure 7 This illustration schematically shows a semantic query process diagram of a data lake governance method according to an embodiment of this application; Figure 8 This illustration schematically shows an access control flow diagram of a data lake governance method according to an embodiment of this application; Figure 9 This illustration schematically shows a metadata integrity restoration process of a data lake governance method according to an embodiment of this application; Figure 10 This schematic diagram illustrates the overall structure of a data lake governance device according to an embodiment of this application. Figure 11 This illustration schematically shows a storage optimization module structure of a data lake governance device according to an embodiment of the present application; Figure 12An exemplary block diagram of a computer program product of a data lake governance method according to an embodiment of this application is shown schematically.

[0019] In the diagram: 1001, Data Acquisition Module; 1002, Formal Context Construction Module; 1003, Concept Lattice Generation Module; 1004, Semantic Embedding Module; 1005, Similarity Calculation Module; 1006, Field Unification Module; 1007, View Construction Module; 1008, Storage Optimization Module; 1009, Priority Scoring Calculation Submodule; 1010, Adaptive Partitioning Submodule; 1011, Hierarchical Compression Submodule; 1201, Computer Program. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0021] Figure 1 The illustration shows a schematic diagram of the unified data view construction process of a data lake governance method according to an embodiment of this application.

[0022] like Figure 1 As shown, the steps include: S101 retrieves raw data from heterogeneous data sources. These heterogeneous data sources can include relational databases, log files, application programming interfaces (APIs), and IoT devices, generating data in various formats including structured, semi-structured, and unstructured types, such as JSON documents, CSV files, Parquet columnar storage formats, and XML files. Raw data is retrieved from these heterogeneous data sources via a Kafka real-time data pipeline. Kafka distributes the retrieved raw data stream using topic and partitioning mechanisms, and the retrieved raw data is stored in a collection area comprised of cloud-native storage.

[0023] S102, constructing a formal context based on the acquired raw data. The formal context includes a set of objects, a set of attributes, and binary relations between objects and attributes; it is an abstract representation of the data structure and its relationships. Mathematically, a formal context can be formalized as a triple. (G,M,I) ,in G It is a set of objects. M It is an attribute set. I G×M It is a binary relation. I It can be represented as a Boolean matrix, where each row corresponds to a data object and each column corresponds to a field attribute. The matrix elements take the value 1 or 0, indicating whether the data object has the corresponding field attribute.

[0024] During formal context construction, Spark is used to parse and scan the collected raw data, extracting all appearing fields and their corresponding data objects. Specifically, the object set is first determined. G and attribute set M In the object set G It contains all datasets or records, with all unique field names included in the attribute set M. Then, a binary relation is constructed. I G×M Wherein the data object g∈G Includes field attributes m∈M hour, I(g,m)=1 ,otherwise I(g,m)=0 This mapping relationship transforms the original data into formalized triples. (G,M,I) This forms the basis for subsequent concept analysis and field unification.

[0025] S103, Generating Concept Lattices Based on Formal Context. A concept lattice is a hierarchical data structure that derives all formal concepts from a constructed formal context and organizes them into a lattice structure according to generalization and specialization relationships. Specifically, conceptual analysis is performed on the formal context. Each formal concept consists of a subset of objects and a subset of their shared attributes. These formal concepts reflect the inherent inclusion and association relationships between field attributes in the original data. Concept lattice generation is performed by computing attribute closures, a process that can discover the largest set of objects corresponding to all attribute combinations. The concept lattice structure formed by this method can identify semantic clusters formed by the co-occurrence of different fields in multiple datasets.

[0026] S104, semantic embedding is performed on the field names of the original data to generate semantic vectors for the field names. To obtain the semantic information of the field names, a pre-trained natural language processing model is used to perform semantic embedding processing on the extracted field names. Specifically, a BERT model based on the Transformer architecture is used as the encoding tool to map each field name into a high-dimensional, dense semantic vector. The generated semantic vector represents the distributed representation of the field name in the semantic space. The numerical features of this semantic vector reflect the potential meaning and functional characteristics of the corresponding field in the context, providing basic data for subsequent semantic similarity calculation.

[0027] (Formula 1) The BERT semantic embedding formula is shown in Formula 1. Field names in the original data w The semantic embedding vector is a high-dimensional quantitative representation of the semantics of a field. This indicates that the field name is processed through a pre-trained BERT model. w Perform semantic encoding operations. This indicates the generated after encoding. d A real number field.

[0028] S105, Calculate the semantic similarity between field names based on semantic vectors. To quantify the degree of semantic association between fields, the semantic similarity between the corresponding semantic vectors of each field is calculated. Specifically, cosine similarity is used as the similarity metric. The degree of semantic similarity is assessed by calculating the cosine of the angle between two semantic vectors in a high-dimensional semantic space. The calculated semantic similarity is represented as a value between -1 and 1, where the closer the value is to 1, the higher the semantic similarity between the two field names. A preset similarity threshold is used to filter the calculation results, automatically identifying field pairs that are highly semantically similar and may represent the same concept.

[0029] (Formula 2) The semantic similarity formula is shown in Formula 2. It is the field name. and Semantic similarity is used to quantify the degree of semantic association between two fields. and These are the semantic embedding vectors corresponding to the two fields, with the numerator being the dot product of the two vectors and the denominator being the sum of the values ​​of the two vectors. The norm product has a value range of [-1, 1]. The closer it is to 1, the more similar the two fields are semantically. It can be used to automatically identify synonymous fields.

[0030] S106. Field names are clustered based on concept lattices and semantic similarity, and a unified field name is generated for each group. Combining the structural association information provided by the concept lattice with the calculated semantic similarity results, clustering is performed on the field names in the original data. The K-Means clustering algorithm is used to group fields with highly similar semantic and structural features into the same group. This clustering process considers both the semantic similarity of the fields and the hierarchical relationship within the concept lattice.

[0031] For each cluster, a unified field name generation process is performed: Firstly, by analyzing the frequency of occurrence, semantic representativeness, and domain relevance of field names within the group, the most representative existing field name is selected as the unified name. Secondly, when there is no clearly representative field within a group, a new field name conforming to business semantics is automatically generated based on the common semantic features of the fields within the group. By determining a unique unified field name for each cluster in this way, data integration barriers caused by naming inconsistencies between different data sources are effectively eliminated, achieving automated unification of field semantics and naming.

[0032] (Formula 3) The K-Means clustering objective function is shown in Equation 3. This is the objective function value of K-Means clustering, representing the compactness of the data within a cluster. The core objective of clustering is to minimize this value. k The number of clusters is dynamically estimated based on the number of data sources and the diversity of fields. Indicates the first i A cluster, x It is a semantic embedding vector of a single field within a cluster. It is the first i The center vector of each cluster It is a vector x With cluster center The squared Euclidean distance can be used to group semantically similar fields into the same cluster, providing a basis for generating unified field names.

[0033] S107. Construct a unified data view based on unified field names. Based on the generated unified field names and corresponding field mapping rules, perform transformation and reorganization operations on the original heterogeneous data. Specifically, original data from different data sources with different field names but the same semantics are structurally aligned and integrated according to a unified field standard, ensuring that data with the same semantics is mapped to a unified field identifier. The constructed unified data view is presented as a logically consistent, globally accessible data representation layer, and this view ensures data consistency through ACID transaction mechanisms.

[0034] Figure 2 The illustration shows a formal context partitioning process diagram of a data lake governance method according to an embodiment of this application.

[0035] like Figure 2 As shown, the steps include: S201: Collect the access frequency of fields, obtain their semantic importance, and calculate the matrix sparsity of formal context. Analyze historical query logs to statistically determine the access frequency of each field, thus obtaining field access frequency data. Simultaneously, based on predefined business rules or through a semantic model, analyze and process the field names and their contexts, assigning each field a semantic importance score that represents its business value.

[0036] For the constructed formal context matrix, sparsity quantification analysis is performed, specifically by calculating the proportion of zero elements in the matrix to the total number of elements to determine the sparsity index. This sparsity index is complementary to the data density, and together they characterize the degree of concentration of effective information in the formal context matrix, providing a data foundation for subsequent partitioning strategies.

[0037] S202 calculates the partition priority score for a field based on access frequency, semantic importance, and matrix sparsity. A predefined decision model is used to comprehensively evaluate the key features of each field to determine its partition priority. Specifically, a lightweight decision tree is used as the decision model to analyze the field's access frequency data, semantic importance score, and the sparsity index of its matrix region. According to a preset strategy, corresponding weights are assigned to each parameter. Through weighted calculation or other mathematical combinations, a corresponding partition priority score is generated for each field. The partition priority score reflects the overall importance of the corresponding field in three dimensions: query popularity characteristics, business value contribution, and storage structure characteristics.

[0038] (Formula 4) The partition priority scoring formula is shown in Formula 4. The partition priority score represents the field's priority during partitioning; a higher score indicates a higher priority for allocating premium storage resources. and It is a preset weight and satisfies , It is based on the historical access frequency of a field obtained by querying log statistics. It is a field semantic importance score generated based on business rules or semantic models. By weighted and integrated access popularity and business value, it provides a clear priority basis for adaptive partitioning.

[0039] S203. Based on the partition priority score, adaptive partitioning is performed on the formal context, and the importance level of each field is determined. Adaptive data partitioning is executed on the formal context matrix based on the calculated partition priority score. Specifically, fields with higher partition priority scores are allocated to fewer storage partitions to reduce storage fragmentation; simultaneously, more parallel processing unit resources are allocated to frequently accessed fields to improve the overall system throughput. Based on the calculated partition priority score, fields are divided into three different importance levels: high, medium, and low. This hierarchy of importance reflects the comprehensive importance of each field in terms of both query popularity and business value contribution.

[0040] Figure 3 The illustration shows a schematic diagram of a formal context hierarchical compressed storage process for a data lake governance method according to an embodiment of this application.

[0041] like Figure 3 As shown, the steps include: S301 employs differentiated compression strategies for storage based on field partitioning at different importance levels. Specifically, for high-importance field partitions, a low-compression-ratio or lossless storage format is used to preserve the original data structure and numerical precision, ensuring that the response performance of high-frequency query operations is not affected. For lower-importance field partitions, a high-compression-ratio algorithm is used, combined with specific techniques such as incremental encoding, to store only key non-zero element information, maximizing storage space efficiency. This differentiated compression mechanism allows data with different value densities to receive storage resource allocation schemes commensurate with their importance. High-value data's fast access capability is fully guaranteed, while the storage cost of low-value data is significantly reduced, thus achieving synergistic optimization of storage efficiency and access performance.

[0042] S302, Construct and store the hierarchical compressed matrix structure. After completing the differentiated compression processing for different importance levels, integrate these compressed partition data to construct a unified hierarchical CSR matrix structure. This hierarchical CSR matrix structure records the compression format, index information, and physical storage location information of data blocks used in different levels of partitions.

[0043] Specifically, for data at the high importance level, the complete CSR storage format is preserved to support fast random access; while for data at the low importance level, Elias encoding is used to incrementally compress the column index, storing only non-zero elements and their position information. The constructed hierarchical compressed matrix structure is persistently stored in the data lake's storage layer, and differentiated memory resources are allocated for different importance levels.

[0044] When executing queries, partition data for high-importance fields is accessed directly from memory, while partition data for low-importance fields is dynamically decompressed using Spark's mapPartitions operation. This storage architecture not only significantly reduces overall memory and storage space usage, but also ensures that the query performance of high-importance data is not affected by compression through a tiered memory allocation strategy, thus achieving synergistic optimization of storage efficiency and access performance.

[0045] Figure 4 The illustration shows a schematic diagram of the knowledge graph construction process of a data lake governance method according to an embodiment of this application.

[0046] like Figure 4 As shown, the steps include: S401, extract metadata information from the aggregation area. The extracted metadata includes technical and administrative attributes, such as the dataset's title, author, creation timestamp, file size, storage format, and its specific path within the data lake. This metadata extraction process is the initial step in the metadata management ETLT workflow, providing foundational data for subsequent construction of attribute graphs and knowledge graphs.

[0047] S402, construct an attribute graph to store metadata information. Nodes represent datasets, and edges represent relationships between datasets. The extracted metadata information is organized into an attribute graph data structure for storage. Specifically, in the constructed attribute graph, each node uniquely corresponds to a dataset entity, and the set of attributes attached to the node fully records the technical and management attributes of the dataset, including metadata information such as title, author, and creation time. Edges in the attribute graph represent semantic relationships between datasets. Edges are classified according to predefined relationship types, including but not limited to citation relationships between datasets, version-derived relationships, and similarity relationships calculated based on content features. Each edge can be assigned a weight attribute to quantify the strength or confidence of the relationship, providing a structured foundation for subsequent knowledge graph construction and semantic querying.

[0048] S403 maps the attribute graph to RDF triples. The constructed attribute graph is transformed and expressed as a set of RDF triples. Specifically, the mapping process follows these rules: each node in the attribute graph represents a dataset entity and is mapped to an RDF subject. The attributes of this node are connected to specific object values ​​through corresponding predicates, thus forming a set of triples describing the dataset. The relationship represented by the edge connecting two nodes in the attribute graph is also mapped to an independent RDF triple, where the subject and object correspond to two related dataset entities, and the predicate explicitly specifies the specific type of the relationship. Through this mapping mechanism, all metadata information and semantic relationships contained in the attribute graph are completely converted into the standard RDF triple format.

[0049] S404 constructs a knowledge graph based on RDF triples. The generated RDF triples are systematically organized and associated to construct a complete and semantically rich knowledge graph. Specifically, triples with the same subject or object are automatically connected, forming a semantic network with data assets as core nodes and various attributes and relationships as edges.

[0050] Figure 5 The illustration shows a schematic diagram of the knowledge graph semantic enhancement process of a data lake governance method according to an embodiment of this application.

[0051] like Figure 5 As shown, the steps include: S501 applies RDFS rules to perform semantic reasoning on the constructed knowledge graph. Based on the predefined semantic rules in the RDF Schema, automated logical reasoning is performed on the constructed knowledge graph. Specifically, by traversing and matching existing RDF triples in the knowledge graph with the pre-defined RDFS rule set, new semantic relationships that conform to the logical rules but are not yet explicitly expressed in the original graph are automatically discovered and generated. The results of the reasoning process constitute a semantic deepening and expansion of the original knowledge graph, enhancing the semantic depth of the metadata.

[0052] S502, based on semantic reasoning, expands the implicit semantic relationships in the knowledge graph. New triples automatically discovered and generated by semantic reasoning are integrated into the existing knowledge graph structure. These new triples represent semantic relationships derived from RDFS rule logic that were not explicitly declared in the metadata, such as inheritance chains of categories, transitive associations of attributes, or equivalence relations of entities. Through this expansion process, the semantic depth and association density of the knowledge graph are significantly enhanced.

[0053] Figure 6 The illustration shows a schematic diagram of the knowledge graph index construction process of a data lake governance method according to an embodiment of this application.

[0054] like Figure 6 As shown, the steps include: S601, based on access frequency and semantic importance, identifies fields and relationship paths in the knowledge graph that require accelerated queries. Access frequency data for each field and relationship path is obtained by analyzing historical query logs, and combined with semantic importance scores generated by predefined business rules or semantic models, a comprehensive evaluation of the query popularity and business value of each data element is conducted. High-frequency accessed and semantically critical data features, including core attribute fields on knowledge graph nodes and frequently traversed semantic relationship paths, are selected as candidate targets for B+ tree index construction, ensuring that index resources are preferentially allocated to the data elements most critical to improving query performance.

[0055] S602, Based on the determined fields and relationship paths, construct a B+ tree index structure. Perform the B+ tree index construction operation according to the selected target fields and relationship paths. Specifically, for key attribute fields on knowledge graph nodes, extract their attribute values ​​to construct an ordered B+ tree index structure to support efficient equality queries and range scan operations. For frequently accessed semantic relationship paths, construct a B+ tree index based on the path's starting node identifier or specific edge attribute values. By optimizing the index key selection strategy, significantly accelerate the node navigation efficiency during graph traversal queries.

[0056] S603 associates the B+ tree index with the nodes and relationships in the knowledge graph for storage. After constructing the B+ tree index structure, each index item is precisely bound to its corresponding specific data element in the knowledge graph. Specifically, for attribute field indexes, a direct pointer mapping is established from the index key to the graph node containing that attribute; for relation path indexes, an association mapping is established from the index key to a series of nodes and edges involved in the path. After the mapping association is completed, the index structure and the knowledge graph data are stored as a unified data structure in the data lake's storage layer.

[0057] Figure 7 The illustration shows a semantic query process diagram of a data lake governance method according to an embodiment of this application.

[0058] like Figure 7 As shown, the steps include: S701 receives and parses query requests for a unified data view. It receives the query request and performs syntax parsing and preliminary semantic extraction on it. Specifically, it identifies the structural features of the query statement through a syntax analyzer, and extracts key elements of the query intent by combining them with a contextual semantic model, including the core entities, attribute features, and relational conditions involved in the query.

[0059] S702 utilizes the B+ tree index built for the knowledge graph to locate metadata nodes and relational paths related to the query request. The parsed query request key is used as the retrieval key to query the constructed and associated B+ tree index structure. Specifically, through the index key-value matching mechanism, the index item matching the query conditions is quickly located in the B+ tree index. Based on the mapping relationship stored in the index item, the corresponding specific metadata node and related semantic relational paths in the knowledge graph are directly accessed, achieving efficient location of the query target.

[0060] S703, based on the expanded semantic relationships in the knowledge graph, performs semantic expansion and context-based retrieval on the query request. After initial indexing, it further utilizes the semantically reasoned and expanded knowledge graph to perform deep semantic mining processing on the query request. Specifically, based on the rich semantic relationship types explicitly declared in the knowledge graph and derived through RDFS rules, including synonyms, hyponyms, and associations, it automatically expands the semantic boundaries of the original query, identifying logically related but literally different potential query targets. Simultaneously, it performs association retrieval operations in the context network of the knowledge graph, analyzing the located nodes and their surrounding connected nodes and derived semantic relationships to obtain a more comprehensive context for the query results.

[0061] S704 integrates the retrieval results from index location and semantic expansion to generate and return semantically rich query results. The results obtained through B+ tree index location and the associated results obtained through semantic expansion are merged and deduplicated. Specifically, the first type of result is precise matching data that is quickly located through the B+ tree index and directly matches the query keywords; the second type of result is contextual information and implicitly related data that are logically strongly relevant to the query intent, retrieved based on explicit and implicit semantic relationships in the knowledge graph. The merged result set is dynamically sorted and hierarchically organized according to a predefined semantic relevance weighting algorithm, ultimately generating and returning a structured query result.

[0062] Figure 8 The diagram illustrates an access control flow of a data lake governance method according to an embodiment of this application.

[0063] like Figure 8 As shown, the steps include: S801 calculates a business importance score based on the business tags and access frequency of a dataset. It obtains predefined semantic tags or tags assigned by the business system from the dataset and determines the semantic importance score of the dataset based on these tags. Simultaneously, it analyzes historical access logs to statistically obtain the access frequency of the dataset over a certain period. A pre-configured linear weight model is used to weight and comprehensively calculate the semantic importance score and access frequency. The calculated business importance score intuitively reflects the overall importance of the dataset from a security and compliance perspective, providing a quantitative basis for subsequent permission level allocation.

[0064] (Formula 5) The formula for scoring the business importance of a dataset is shown in Formula 5. It is a business importance score for datasets, used to quantify the security and business value of datasets, providing a core basis for permission allocation. and It is a preset weight. f It is the historical access frequency of the dataset obtained through log statistics. It is a semantic importance score of datasets generated based on RDF knowledge graphs. By weighted fusion of access frequency and semantic importance, it accurately quantifies the overall importance of the dataset.

[0065] S802 generates access permission levels and rules based on business importance scores and semantic tags. The calculated business importance score is mapped to a specific access permission level. The mapping process, based on a preset threshold range, divides permission levels into three levels: public, authorized, and restricted. Higher scores correspond to stricter permission level constraints, thus achieving differentiated control over data of varying value and sensitivity. Furthermore, it combines the semantic tags of the dataset generated through an RDF knowledge graph to generate access control rules based on permission levels, semantic tag sets, and contextual conditions.

[0066] S803, based on access permission rules, enforces access control over datasets. It extracts key contextual information from access requests, such as user identity, access goal, operation type, and time, and automatically matches and logically evaluates this information against pre-generated, structured access permission rules. Based on the highest priority or most specific rule found, it makes an immediate access control decision, allowing or denying the access request.

[0067] S804 monitors access behavior to the dataset and detects abnormal access based on a time series model. It collects and records all user and application access events to the dataset, forming a detailed access behavior log. These logs are converted into time series data, characterized by the time series of access frequency, operation time intervals, and time series changes in the amount of data accessed by specific users or roles. A dynamic behavior prediction model is built by learning from historical normal access patterns using a Long Short-Term Memory (LSTM) network. The model calculates the absolute deviation between the predicted and actual values ​​and compares it to a preset deviation threshold; if the absolute deviation exceeds the preset threshold, the access behavior is considered abnormal.

[0068] S805: When abnormal access is detected, a log is recorded, and a dynamic evaluation and adjustment of the access permission rules for the relevant dataset is triggered. All contextual details of the detected abnormal access event, including the access timestamp, user identity, operation type, target dataset identifier, and abnormal score, are fully recorded in a dedicated audit log. Simultaneously, a dynamic re-evaluation process for the access permission rules of the dataset involved in the abnormal access is triggered. During the re-evaluation, the latest access behavior pattern data, the dynamic changes in the dataset's business importance score, and the update status of semantic tags are analyzed to recalculate and dynamically adjust the access permission level and access control rules corresponding to the dataset.

[0069] Figure 9 The illustration shows a schematic diagram of the metadata integrity restoration process of a data lake governance method according to an embodiment of this application.

[0070] like Figure 9 As shown, the steps include: S901 assigns basic weights to the rules for assessing metadata integrity. Rules for assessing dataset metadata integrity are retrieved from a predefined metadata governance rule base. Each rule defines a specific quality requirement that metadata must meet; these rules collectively constitute the benchmark for metadata quality verification. Based on the dataset's business tags, an initial basic weight value is assigned to each rule. This weight value, between 0 and 1, reflects the relative importance or stringency of the rule in a general scenario.

[0071] S902, dynamically adjust the weights of corresponding rules based on the business importance score of the dataset. Based on the calculated business importance score of the dataset, adjust the allocated base weights. Specifically, for critical datasets with high business importance scores, increase the weight coefficients of their core integrity rules, making the satisfaction of these rules have a greater impact on the overall integrity score, thus reflecting the strict quality control requirements for high-business-value data. For datasets with low business importance scores, decrease the weight coefficients of certain non-core integrity rules, effectively avoiding excessive investment in data governance resources. This step ensures that the rigor of data governance matches the business importance of the data itself, achieving an optimal balance between the efficiency of governance resource investment and the level of data quality assurance.

[0072] S903, Calculate the weighted integrity score of the dataset based on the adjusted rule weights. Specifically, examine the actual satisfaction of each integrity rule on the dataset's metadata, generating a binary rule satisfaction result, where 0 indicates non-satisfaction and 1 indicates satisfaction. Multiply the satisfaction result of each rule by its corresponding dynamic weight value to obtain the weighted score for that rule. Finally, by summing the weighted scores of all rules, calculate the overall weighted integrity score of the dataset, which reflects the comprehensive quality level of the dataset's metadata.

[0073] (Formula 6) The formula for weighted integrity scoring of metadata is shown in Formula 6. It is a weighted integrity score for metadata, used to quantify the degree to which metadata meets quality standards. n It is the total number of metadata integrity rules. i It is a rule index. It is the first i The dynamic weights of the rules are used to determine the scoring results, which range from [0,1].

[0074] S904: When the weighted integrity score falls below a preset threshold, a metadata repair task for that dataset is triggered. The calculated weighted integrity score of the dataset is compared and analyzed with a pre-configured quality pass threshold. This quality pass threshold is a dynamically configurable parameter representing the minimum integrity standard that the dataset's metadata must meet. When it is determined that the weighted integrity score of a dataset is below this quality pass threshold, a standardized metadata repair task process for that dataset is triggered to improve its metadata quality to a passable level.

[0075] S905. After the metadata repair task is executed, the weighted integrity score of the dataset is recalculated and updated. Following the metadata repair task, the metadata content of the relevant dataset has changed. To accurately assess the repair effect and reflect its latest quality status, an integrity assessment process is implemented for the dataset. Based on a rule base and dynamic weights, the updated metadata after repair is re-verified and recalculated to obtain a new weighted integrity score. This weighted integrity score will be used to determine whether the repaired metadata meets the quality requirements.

[0076] S906. The updated metadata is synchronized to the knowledge graph, completing the repair process. Specifically, this synchronization process includes updating the attribute values ​​of relevant nodes in the knowledge graph and, when necessary, adding or correcting semantic relationship edges between nodes to ensure that the knowledge graph accurately reflects the latest and most accurate state of the data assets.

[0077] Figure 10 The schematic diagram illustrates the overall structure of a data lake governance device according to an embodiment of this application.

[0078] like Figure 10 As shown, the device includes a data acquisition module 1001, a formal context construction module 1002, a concept lattice generation module 1003, a semantic embedding module 1004, a similarity calculation module 1005, a field unification module 1006, a view construction module 1007, and a storage optimization module 1008.

[0079] The data acquisition module 1001 is configured to acquire raw data from heterogeneous data sources through Kafka real-time data pipelines. The heterogeneous data sources include relational databases, log files, application programming interfaces and IoT devices. Kafka distributes the acquired raw data streams through topic and partitioning mechanisms, and the acquired raw data is stored in a collection area composed of cloud-native storage.

[0080] Formal context construction module 1002 is configured to construct a formal context based on the acquired raw data. The formal context is defined as a triple. (G,M,I) ,in GIt is a set of objects. M It is an attribute set. I It is a binary relation. Spark parses and scans the original data to extract all the fields that appear and the data objects to which they belong, and constructs a Boolean matrix representing the relationship between the fields and the data objects.

[0081] The concept lattice generation module 1003 is configured to generate concept lattices based on formal context. It discovers the largest set of objects corresponding to all attribute combinations by computed attribute closures, forming a hierarchical data structure. Each formal concept is composed of a subset of objects and a subset of attributes that they share.

[0082] The semantic embedding module 1004 is configured to perform semantic embedding on the field names of the original data. It uses the BERT model based on the Transformer architecture to map the field names into high-dimensional, dense semantic vectors to capture the distributed representation of the field names in the semantic space.

[0083] The similarity calculation module 1005 is configured to calculate the semantic similarity between field names based on semantic vectors. It uses cosine similarity as the metric to calculate the cosine value of the angle between two semantic vectors in the high-dimensional semantic space and generate a similarity value between -1 and 1.

[0084] The field unification module 1006 is configured to cluster field names based on concept lattice and semantic similarity, and generate unified field names for each group. It uses the K-Means clustering algorithm to group fields with highly similar semantic and structural features into the same group, and selects the most representative field names or generates unified field names by analyzing the frequency of occurrence, semantic representativeness and domain relevance of field names within the group.

[0085] The view building module 1007 is configured to build a unified data view based on a unified field name. It performs transformation and reorganization operations on the original heterogeneous data according to the unified field name and the corresponding field mapping rules, and performs structural alignment and data integration on data with different field names but the same semantics according to a unified field standard.

[0086] Storage optimization module 1008 is configured to partition and compress the storage of the formal context.

[0087] Figure 11 The illustration shows a schematic diagram of the storage optimization module structure of a data lake governance device according to an embodiment of this application.

[0088] like Figure 11 As shown, the storage optimization module 1008 includes a priority scoring calculation submodule 1009, an adaptive partitioning submodule 1010, and a hierarchical compression submodule 1011.

[0089] Among them, the priority score calculation submodule 1009 is configured to obtain the access frequency data of the field by analyzing the historical query logs, and combine it with the semantic importance score generated by the predefined business rules or semantic model, and use a lightweight decision tree model to calculate the partition priority score.

[0090] The adaptive partitioning submodule 1010 is configured to divide fields into three importance levels—high, medium, and low—based on partition priority scores. It allocates high-importance fields to fewer storage partitions to reduce storage fragmentation, while allocating more parallel processing unit resources to fields with high access frequency to improve the overall system computing throughput.

[0091] The hierarchical compression submodule 1011 is configured to use differentiated compression strategies for partitions of different importance levels. For fields of high importance, the complete CSR storage format is preserved to support fast random access. For fields of low importance, Elias encoding is applied to incrementally compress the column index, storing only non-zero elements and their position information. Low importance fields are dynamically decompressed through Spark's mapPartitions operation, thereby significantly reducing the overall storage space usage while ensuring the query performance of high importance data.

[0092] Figure 12 An exemplary block diagram of a computer program product of a data lake governance method according to an embodiment of this application is shown schematically.

[0093] like Figure 12 As shown, the computer program product stores a computer program 1201, which, when executed by a processor, implements the method provided in any embodiment of this application.

[0094] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0095] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0096] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0097] It should also be noted that in the system and method of this application, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of this application.

[0098] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this application is not limited to the specific aspects of the processes, machines, manufacturing, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufacturing, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described above can be utilized. Therefore, the appended claims include such processes, machines, manufacturing, events, means, methods, or actions within their scope.

[0099] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0100] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A data lake governance method, characterized in that, include: Obtain raw data from heterogeneous data sources; A formal context is constructed based on the acquired raw data; wherein, the formal context includes a set of objects, a set of attributes, and binary relations between objects and attributes; Generate a concept lattice based on the formal context; Semantic embedding is performed on the field names of the original data to generate semantic vectors for the field names; Calculate the semantic similarity between the field names based on the semantic vector; Based on the concept lattice and the semantic similarity, the field names in the original data are clustered and grouped, and a unified field name is generated for each group; A unified data view is constructed based on the unified field names.

2. The data lake governance method according to claim 1, characterized in that, Partitioning and compressing the formal context for storage includes: Calculate the partition priority score based on the access frequency and semantic importance of the field name; The formal context is partitioned based on the partition priority score; The formal context after partitioning is compressed and stored.

3. The data lake governance method according to claim 1, characterized in that, Metadata management with semantic enhancement for the unified data view includes: Extract metadata information of the dataset in the unified data view, wherein the metadata information includes at least one of title, author, creation time and storage path; A property graph is constructed to store the metadata information; wherein nodes represent the datasets and edges represent the relationships between the datasets. The attribute graph is mapped to resource description framework triples to construct a knowledge graph; The implicit relationships in the knowledge graph are extended based on RDF schema rules.

4. The data lake governance method according to claim 3, characterized in that, Construct a B+ tree index for the knowledge graph; In response to a query request, a semantic query is performed based on the indexed knowledge graph and the expanded implicit relationships.

5. The data lake governance method according to claim 2 or 3, characterized in that, Access control for the unified data view includes: A business importance score is calculated based on the business tags and access frequency of the dataset in the unified data view; Access control rules are dynamically generated based on the business importance score and the semantic labels of the dataset. Monitor access behavior to the unified data view and detect abnormal access based on the sequence of access behavior; When the abnormal access is detected, the abnormal access is recorded in the log, and the access permissions are adjusted according to the access control rules.

6. The data lake governance method according to claim 5, characterized in that, Governing the integrity of the metadata in the unified data view includes: Weights are assigned to the rules for assessing the integrity of the metadata; Based on the business importance score of the dataset, adjust the weights of the corresponding rules and calculate the weighted integrity score; If the weighted integrity score is lower than a preset threshold, a metadata repair task for the dataset is triggered.

7. A data lake governance device, characterized in that, include: The data acquisition module is configured to acquire raw data from heterogeneous data sources; The formal context construction module is configured to construct a formal context based on the acquired original data; wherein, the formal context includes an object set, an attribute set, and a binary relationship between objects and attributes; The concept lattice generation module is configured to generate concept lattices based on the formal context. The semantic embedding module is configured to perform semantic embedding on the field names of the original data and generate semantic vectors for the field names; The similarity calculation module is configured to calculate the semantic similarity between the field names based on the semantic vector; The field unification module is configured to cluster the field names in the original data according to the concept grid and the semantic similarity, and generate a unified field name for each group. The view building module is configured to build a unified data view based on the unified field name.

8. The data lake governance device according to claim 7, characterized in that, It also includes a storage optimization module, which includes: The priority score calculation submodule is configured to calculate the partition priority score based on the access frequency and semantic importance of the field name; An adaptive partitioning submodule is configured to partition the formal context based on the partition priority score; The hierarchical compression submodule is configured to compress and store the partitioned formal context.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the data lake governance method according to any one of claims 1 to 6.

10. A computer program product comprising a computer-readable storage medium on which a computer program is stored, characterized in that, When the computer program is executed by the processor, it implements the data lake governance method according to any one of claims 1 to 6.

Images