Data lake metadata management method based on semantic synthesis and text vectorization

Abstract

The invention relates to the technical field of natural language processing, in particular to a semantic synthesis and text vectorization-based data lake metadata governance method, which comprises the following steps of: performing vector coding of field content keywords and adjacent context words on the basis of field names, data types and description information of data sources in a data lake; and extracting semantic co-occurrence groups of the fields in the differentiated contexts, identifying distances between semantic vectors, and judging whether the distances are in a similar field grouping range or not to obtain a field semantic collection quantity. According to the method, a semantic co-occurrence relation is extracted through field keywords and context vector coding, a semantic hierarchical structure is constructed, the field division precision and abstract ability are improved, the consistency is judged in combination with a context semantic structure, a high-frequency alternative path is adjusted to optimize a semantic structure, and word vector similarity and a structure retention rate are fused to realize field merging; the continuity and the accuracy of a treatment structure are improved, and the intelligence and the consistency of metadata treatment in the data lake are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly to a method for governing data lake metadata based on semantic synthesis and text vectorization. Background Art

[0002] The technical field of natural language processing includes technologies for a computer to process, analyze, understand, and generate text data. The core content of this field mainly involves technical means such as text vectorization, semantic understanding, automatic reasoning, and text generation. Natural language processing technology aims to simulate and analyze human language through a computer, enabling the computer to understand, generate, and process text data. The technical methods in this field are widely applied in multiple directions such as information retrieval, machine translation, sentiment analysis, speech recognition, and text generation. By adopting various algorithms and models, especially deep learning-based models, natural language processing technology can effectively process and intelligently analyze a large amount of unstructured text data.

[0003] Among them, the data lake metadata governance method refers to a technology for managing and governing metadata in a data lake through semantic synthesis and text vectorization techniques. This method aims at how to effectively manage large-scale heterogeneous data stored in a data lake environment, especially for quality control of metadata and improvement of usage efficiency. By adopting text vectorization technology, metadata is transformed into a numerical vector form, which is convenient for semantic analysis and reasoning. By adopting semantic synthesis methods, metadata is understood, summarized, and deduced to ensure that the data in the data lake can be efficiently queried, accessed, and managed. The solution to this patent theme involves structuring the metadata in the data lake, accurately classifying and optimizing the storage of metadata based on semantic understanding methods, and achieving data governance goals through text vectorization.

[0004] When dealing with data lake metadata, the prior art relies on static vectorized representations of metadata and preset semantic classification methods, making it difficult to effectively capture the dynamic semantic commonalities of fields in diverse contexts. This leads to one-sidedness in field attribution judgment. For example, the actual meaning differences of fields with the same name in different data sources are not accurately identified, easily causing classification errors. The existing methods lack a dynamic judgment mechanism for the semantic hierarchical relationships of fields. The semantic structure hierarchy division is often based on fixed rules, making it difficult to handle the hierarchical drift problem in the context of complex semantic evolution. The determination of field consistency usually relies on field name or path similarity, failing to comprehensively consider semantic dependency relationships, resulting in a high cross-data-source field conflict rate. During the structure optimization process, the prior art has a lag in responding to path changes and lacks the ability to dynamically identify high-frequency alternative paths. The field semantic paths cannot adapt to the rhythm of metadata evolution, weakening the system's adaptability. During the field merging process, there is a lack of a linkage judgment standard for similarity and structure retention rate, easily causing semantic drift after field merging, thus affecting the accuracy and consistency of the governance structure and restricting the improvement of metadata governance capabilities in the overall data lake environment. Summary of the Invention

[0005] An object of the present invention is to solve the shortcomings existing in the prior art and propose a data lake metadata governance method based on semantic composition and text vectorization.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A data lake metadata governance method based on semantic composition and text vectorization, including the following steps: S1: Based on the field names, data types, and description information of data sources in the data lake, perform vector encoding on the field content keywords and adjacent context words, extract the semantic co-occurrence groups of fields in different contexts, identify the distances between semantic vectors, determine whether they are within the range of the same type of field grouping, and obtain the field semantic aggregation quantity; S2: Based on the field semantic aggregation quantity, extract the semantic center vectors of the field attribution groups, compare the cosine angle values of the semantic center vectors between field groups, and determine whether the angle values are higher than the field semantic level to obtain the field semantic hierarchical structure; S3: Invoke the field semantic hierarchical structure, extract the context semantic structures, semantic paths, and semantic dependency relationships between fields in different data sources, determine whether they exceed the set consistency judgment range, and if so, mark the fields as conflict items to obtain the metadata consistency conflict rate; S4: Based on the metadata consistency conflict rate, collect the alternative field paths in the data lake metadata change log, extract the occurrence frequencies of the replaced fields in the original paths, screen the high-frequency alternative paths for structure position replacement to obtain the semantic path adjustment data.

[0007] As a further solution of the present invention, the field semantic aggregation quantity includes a field attribution label, a context semantic feature, and a vector aggregation index. The field semantic hierarchical structure includes a hierarchical grouping identifier, an inter-semantic layer mapping, and a field group hierarchical division. The metadata consistency conflict rate includes a conflict field ratio, a field conflict distribution, and a semantic conflict level. The semantic path adjustment data includes a field replacement path, a structure adjustment record, and a replacement frequency index.

[0008] As a further solution of the present invention, the specific steps for obtaining the field semantic aggregation quantity are as follows: S111: Based on the field name, data type, and description information of the data source in the data lake, extract the keywords and context in the field, and analyze the word frequency and combination structure through vectorized encoding to obtain the field local semantic vector; S112: Call the field local semantic vector, identify the Euclidean distance and cosine similarity according to the semantic vectors of the field in different contexts, perform a distance threshold judgment on similar fields, and use the formula: ; Calculate the field semantic matching degree value, compare the matching degree value with the preset semantic attribution benchmark range, group and merge the field vectors that meet the benchmark to obtain the field semantic attribution grouping quantity; Among them, represents the field semantic matching degree value, represents the field and the field of the local semantic vector, represents the field and the field in the local semantics of the semantic energy value, is the cosine similarity value between the semantic vectors of the field and the field ; S113: Based on the field semantic attribution grouping quantity, for the fields that have not been attributed, judge whether the semantic matching degree value with the fields in the attribution grouping meets the attribution conditions, and retain the fields that do not meet the attribution conditions as independent fields to obtain the field semantic aggregation quantity.

[0009] As a further solution of the present invention, the specific steps for obtaining the field semantic hierarchical structure are as follows: S211: Based on the field semantic aggregation quantity, extract the semantic center vector of each group of field attribution groups, calculate the cosine included angle value between the semantic center vectors of the field groups, screen the field groups with a cosine included angle value lower than the preset field semantic level threshold, and obtain the preliminary semantic similarity between the field groups; S212: Invoke the preliminary semantic similarity between field groups, determine whether it is higher than the field semantic hierarchy division standard value, identify the field groups that meet the standard, merge them into the same hierarchical group, and use the formula: ; Obtain the field semantic hierarchy merging value; Among them, represents the field semantic hierarchy merging value, represents the field at the semantic center component of the dimensional vector, represents the field at the semantic center component of the dimensional vector, represents the number of vector dimensions; S213: Invoke the field semantic hierarchy merging value, organize the merged field hierarchical group, analyze the hierarchical structure mapping relationship, and obtain the field semantic hierarchical structure.

[0010] As a further solution of the present invention, the steps for obtaining the metadata consistency conflict rate are specifically as follows: S311: Invoke the semantic hierarchy field group defined in the field semantic hierarchical structure, extract the context semantic structure of the field in the differential data source, parse the semantic path, and obtain the field semantic dependency matrix; S312: According to the field semantic dependency matrix, judge the overlap of the field semantic paths, combine the field context semantic structure, identify the field semantic consistency, and use the formula: ; Calculate the field consistency deviation value, screen the field pairs that exceed the set range, and obtain the field consistency exception set; Among them, represents the field consistency deviation value, represents the context semantic feature of field k, represents the context semantic feature of the comparison field k, represents the weight coefficient of field k in the semantic path, and p represents the total number of fields in the field group; S313: Based on the field consistency exception set, invoke the marked fields, determine the semantic path deviation situation, screen the fields that exceed the judgment range, and perform consistency conflict marking to obtain the metadata consistency conflict rate.

[0011] As a further solution of the present invention, the steps for obtaining the semantic path adjustment data are specifically as follows: S411: Extract the metadata change log of the data lake based on the metadata consistency conflict rate, identify the occurrence frequencies of the alternative field paths and their corresponding original paths, analyze the substitution frequencies in the change log in combination with the occurrence times of the original path fields, filter the paths with substitution frequencies higher than the set threshold, and obtain the data of the paths with high substitution frequencies; S412: Based on the data of the paths with high substitution frequencies, analyze the substitution position adjustment degree of each path in the data lake metadata, judge the adaptability and position of the alternative path in the structure, and use the formula: ; Obtain the structure position fitness data; Among them, represents the structure position fitness data, represents the substitution frequency of the path , represents the occurrence frequency of the path in the original path, represents the total number of records in the change log, represents the log number when the alternative path first appears in the change log, represents the log number when the original path first appears in the change log, represents the total number of paths; S413: According to the structure position fitness data, filter the field paths in the alternative paths, replace the corresponding field positions of the original paths, and obtain the semantic path adjustment data.

[0012] As a further solution of the present invention, the method further includes step S5: S5: Based on the semantic path adjustment data, extract the word vector values of the fields in the currently collected data, compare the fusion similarity and grouping retention rate of the field word vectors before and after adjustment, merge the qualified fields into a unified structure group, and obtain the metadata governance fusion structure; The metadata governance fusion structure includes a fusion field group, a unified semantic structure, and a word vector similarity matrix.

[0013] As a further solution of the present invention, the steps for obtaining the metadata governance fusion structure are specifically: S511: Based on the semantic path adjustment data, extract the fields in the currently collected data, identify the word vector values of each field, analyze the semantic relationships of the fields in the differential data tables, and obtain the field word vector data; S512: Invoke the field word vector data, compare the fusion similarity of the field word vectors before and after adjustment, analyze the fusion matching degree of the fields after adjustment with the original structure, identify the grouping retention rate, and use the formula: ; Calculate the field fusion similarity data, analyze the similarity data, screen the fields that meet the grouping threshold, and obtain the field grouping retention rate data; Among them, represents the field fusion similarity data, and represent the field word vector values before and after adjustment respectively, represents the hierarchical depth of the field in the data table, represents the number of references of the field in the data lake, represents the number of structural changes of the field; S513: Based on the field grouping retention rate data, screen the eligible fields, merge them into a unified structure group, and obtain the metadata governance fusion structure.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In the present invention, by performing vector encoding on the keywords and context of the fields in the data source, identifying the semantic co-occurrence relationship and using the semantic distance as the attribution criterion, the recognition depth of the semantic relationship of the fields in different contexts is improved, the accuracy of the field attribution division is enhanced, and a hierarchical structure is established by combining the cosine angle relationship of the semantic center vectors of each attribution group, effectively constructing a semantic abstraction and aggregation system between fields, solving the problem of fuzzy semantic hierarchy of fields in the traditional structure, further analyzing the semantic context structure and dependency relationship of fields in multi-source data, accurately determining field consistency, and improving the ability of metadata quality control in a multi-source heterogeneous environment. Adjust the structural position of the high-frequency alternative field path in the change log to achieve dynamic optimization of the field semantic path, promote the enhancement of the adaptive ability of the semantic structure in the actual business change, and based on the comparison between the path adjustment result and the word vector in the currently collected data, the fusion similarity and the structure retention rate are comprehensively judged to ensure the accuracy of the field merging operation and the stability of the system structure, improve the coherence and accuracy of metadata governance in the process of structural evolution, and significantly improve the intelligent level and processing efficiency of metadata management in a complex data lake environment through multi-dimensional semantic comparison and dynamic merging strategy, enhancing the visibility at the semantic level and the consistency of data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of the main steps of the present invention; Figure 2 is a flowchart for obtaining the semantic collection quantity of fields in the present invention; Figure 3 is a flowchart for obtaining the semantic hierarchical structure of fields in the present invention; Figure 4 is a flowchart for obtaining the metadata consistency conflict rate in the present invention; Figure 5Flowchart for obtaining semantic path adjustment data in the present invention; Figure 6 Flowchart for obtaining the metadata governance fusion structure in the present invention. Detailed implementation manners

[0016] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the present invention. In addition, in the description of the present invention, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0018] Embodiment 1

[0019] Please refer to Figure 1 , the present invention provides a technical solution: a data lake metadata governance method based on semantic composition and text vectorization, including the following steps: S1: Based on the field names, data types, and description information of data sources in the data lake, perform vector encoding on the keywords and adjacent context words in the field content, extract the semantic co-occurrence groups of fields in different contexts, identify the corresponding distances between the semantic vectors of each group of fields, and determine whether the semantic distance values are within the range of the same type of field grouping. If satisfied, the fields are classified into the same field attribution group; otherwise, they are retained as independent fields to obtain the field semantic aggregation quantity; S2: Based on the field semantic aggregation quantity, extract the semantic center vectors of each group of field attribution groups, compare the cosine angle values of the semantic center vectors between field groups, determine whether the angle values are higher than the field semantic level, and divide the judgment standard values. If satisfied, they are merged into the same hierarchical group to obtain the field semantic hierarchical structure; S3: Call the semantic level field groups defined in the field semantic hierarchical structure, extract the context semantic structures, semantic paths, and semantic dependencies between fields in different data sources, and determine whether they exceed the set field consistency judgment range. If exceeded, mark the fields as conflict items to obtain the metadata consistency conflict rate; S4: Based on the metadata consistency conflict rate, collect the alternative field paths in the metadata change log of the data lake, extract the occurrence frequency of the replaced fields in the original paths, filter the paths with high replacement frequencies for structural position replacement, and obtain the semantic path adjustment data; S5: Based on the semantic path adjustment data, extract the word vector values of the fields in the currently collected data, compare the fusion similarity and grouping retention rate of the field word vectors before and after adjustment, and merge the eligible fields into a unified structure group to obtain the metadata governance fusion structure.

[0020] The field semantic aggregation quantity includes field attribution labels, context semantic features, and vector aggregation metrics. The field semantic hierarchical structure includes hierarchical grouping identifiers, inter-semantic layer mappings, and field group hierarchical divisions. The metadata consistency conflict rate includes the proportion of conflict fields, field conflict distribution, and semantic conflict levels. The semantic path adjustment data includes field replacement paths, structural adjustment records, and replacement frequency metrics. The metadata governance fusion structure includes fusion field groups, unified semantic structures, and word vector similarity matrices.

[0021] Please refer to Figure 2 , and the specific steps for obtaining the field semantic aggregation quantity are as follows: S111: Based on the field names, data types, and description information of the data sources in the data lake, extract the keywords and context in the fields, and analyze the word frequency and combination structure through vectorization encoding to obtain the local semantic vectors of the fields; In the process of semantic analysis of the fields in the data lake, identify the keywords and related context words in each field. For example, when processing sales data, the keywords are "sales amount" and "region", and the context words cover descriptive words such as "increase" and "decrease". The selection of words is based on the business meaning of the fields and the actual content of the data. Through efficient vectorization encoding techniques, the words are converted into a mathematically processable vector form. For example, using word embedding techniques such as Word2Vec, each word is transformed into a multi-dimensional numerical vector. The importance of this step is to lay the foundation for subsequent semantic analysis, ensuring that the vector of each word can reflect the characteristics of its semantic content. Based on the vectors, construct the local semantic vector matrix of each field. This matrix is calculated based on the word frequency and word vectors within the field, and its result will directly affect the accuracy of field classification. Finally, generate the local semantic vectors of the fields. The result provides a quantitative perspective to understand and compare the semantic similarities of different fields.

[0022] S112: Invoke the local semantic vectors of the fields. According to the semantic vectors of the fields in different contexts, identify the Euclidean distance and cosine similarity, and perform distance threshold judgment for similar fields. Use the formula: ; Calculate the semantic matching degree value of the calculated field, compare the matching degree value with the preset semantic attribution benchmark range, group and merge the field vectors that meet the benchmark, and obtain the field semantic attribution grouping quantity; Among them, represents the semantic matching degree value of the field, represents the field and the local semantic vector of the field ; represents the field and the semantic energy value of the field in the local semantics; is the cosine similarity value between the semantic vectors of the field and the field ; For the local semantic vector groups of the field and the field in different context scenarios, it is necessary to detect the semantic matching degree value between the two fields. During the execution process, first collect the local semantic vectors of the field and the field . Assume that the vector of the field is , and the vector of the field is ; First, calculate the squared Euclidean distance, that is ; Calculate the semantic energy value , using the sum of the squares of the vector elements, to get , similarly ; Then calculate the cosine similarity value , by dividing the dot product of the vectors by the product of the norms. The dot product of the vectors is , the norms are respectively , and the cosine similarity is ; Substitute the values as: ; Set the semantic attribution benchmark range to 0.05, the value , meets the attribution condition, and group the field and the field into the same group. Repeat the above calculation process, traverse all field combinations and classify and merge them to obtain the field semantic attribution grouping quantity; The benefit of the formula is that by combining the squared Euclidean distance with the square root of the semantic energy and introducing the product of the cosine similarity at the same time, the matching degree value comprehensively reflects the semantic differences and similarities between fields, so as to more reasonably distinguish the field attribution relationship. The result shows that the field and the field Belong to the same field group and are further used to generate the field semantic attribution grouping quantity.

[0023] S113: Based on the field semantic attribution grouping quantity, for the fields that have not been attributed, determine whether the semantic matching degree value with the fields in the attribution group meets the attribution condition, and retain the fields that do not meet the attribution condition as independent fields to obtain the field semantic aggregation quantity; In the process of determining whether a field is retained as an independent field, an important step is to use the previously calculated field semantic attribution grouping quantity as a reference to check those fields that have not been classified. For example, a field with a unique semantics did not match any existing group in the previous step. For example, a field related to sales in a specific region failed to form a semantic connection with the fields due to the particularity of the vocabulary (such as place names). Conduct a re-evaluation of the semantic matching degree of the field to see if it can meet the matching criteria of a certain existing group. The evaluation is based on the semantic distance calculation formula used in the previous step. If the matching degree is lower than the set threshold, the field is retained as an independent field. This ensures that each field is properly classified, either by joining an existing group or existing as a new independent entity, and finally obtains the field semantic aggregation quantity, providing an efficient data organization method for the management and optimization of the data lake.

[0024] Please refer to Figure 3 , the steps for obtaining the field semantic hierarchical structure are specifically as follows: S211: Based on the field semantic aggregation quantity, extract the semantic center vector of each group of field attribution groups, calculate the cosine included angle value of the semantic center vectors between the field groups, and screen out the field groups with a cosine included angle value lower than the preset field semantic level threshold to obtain the preliminary semantic similarity between the field groups; First, based on the metadata information in the existing data lake, extract the data characteristics of each field group to form an initial semantic center vector. The process involves evaluating the data type, usage frequency, and association degree of each field. After a series of data processing steps, such as cleaning and standardization, use vectorization technology to convert the field characteristics into numerical semantic vectors. For example, the fields of a customer data table include name, address, transaction frequency, etc. By evaluating the correlation between the fields and customer classification, the semantic vectors of each field can be initially established. Then, calculate the cosine included angle value between each field vector to determine the similarity, which is crucial for subsequent data management and query optimization. Through actual data examples, if the cosine included angle value between the customer name and the customer address is high, it means that these two fields are semantically similar and belong to the same semantic level. For such fields, obtain the preliminary semantic similarity between the field groups.

[0025] S212: Invoke the preliminary semantic similarity between field groups, determine whether it is higher than the field semantic level division standard value, identify the field groups that meet the standard, merge them into the same level group, and use the formula: ; Obtain the field semantic level merging value; where, represents the field semantic level merging value, represents the field at the semantic center component of the dimensional vector, at the semantic center component of the dimensional vector;

[0026] Determine whether the similarity is higher than the preset field semantic level division standard, which needs to be set according to the actual business scenario. For example, in the user behavior analysis of an e-commerce platform, although the user's click behavior and purchase behavior may seem different on the surface, from the perspective of data analysis, they can have a high semantic correlation. By setting a reasonable judgment threshold, seemingly different but actually related behaviors can be classified into the same level. For example, in a large e-commerce platform, the fields "user click count" and "user purchase count" respectively represent the number of times a user clicks on a product and the number of times the user finally purchases a product within a certain time window. The fields can be vectorized as: , , when calculating the cosine angle value between field groups; First, calculate the numerator part: ; Then, calculate the two parts of the denominator: ; ; Calculate the cosine angle value: ; Then, calculate the mean deviation of the second part: ; Finally, calculate the merging value: ; Assume that the field semantic level division standard value is set to 1.8, then is higher than the set threshold. Therefore, the two fields "user click count" and "user purchase count" should belong to the same level group and be further merged. In this way, data can be classified and managed more effectively, and finally, the semantic level merging relationship of the field group is analyzed to obtain the field semantic level merging value.

[0027] S213: Call the merged value of the field semantic levels, organize the merged field level groups, analyze the hierarchical structure mapping relationship, and obtain the field semantic hierarchical structure; Organize the merged field level groups to establish a more refined hierarchical structure mapping relationship. This step is to better utilize data to support decision-making in practical applications. For example, in financial risk management, through hierarchical structured management of different types of transaction data, potential risk events can be quickly identified and responded to. Through actual data cases, such as hierarchical processing of transaction amount and transaction time data, it can help the risk management department locate abnormal transaction behaviors faster, thereby implementing effective risk control. Through such processing, the field semantic hierarchical structure can be obtained.

[0028] Please refer to Figure 4 , the steps for obtaining the metadata consistency conflict rate are specifically as follows: S311: Call the semantic level field group defined in the field semantic hierarchical structure, extract the context semantic structure of the field in different data sources, parse the semantic path, and obtain the field semantic dependency matrix; In a data lake environment, performing field semantic analysis for different data sources is a key step in improving data quality management. For example, in e-commerce and medical information, accurate division of field semantic levels and establishment of semantic dependency relationships can effectively improve the accuracy and efficiency of data integration. The first step based on the call of the semantic hierarchical structure involves extracting the context semantic structure of fields from multiple data sources, which includes analyzing the usage frequency of field data, associated data types, and dependencies between fields. For example, in e-commerce, for product data and user review data, by extracting the associated usage of product IDs and user IDs, a dependency graph between fields can be constructed, analyzing how the semantic path of field data supports data consistency between data sources. Through specific case calculations, such as the matching degree analysis of product data and user behavior data, it can clearly show how to optimize the data integration process based on the semantic dependency relationship of fields, and ultimately obtain the field semantic dependency matrix, which will directly affect the efficiency and accuracy of subsequent data integration and analysis. This methodology based on actual data analysis not only improves the transparency of data processing but also provides empirical support for data governance in the data lake.

[0029] S312: According to the field semantic dependency matrix, judge the overlap situation of the field semantic paths, combine the context semantic structure of the fields, identify the field semantic consistency, and use the formula: ; Calculate the field consistency deviation value, filter out the field pairs that exceed the set range, and obtain the field consistency anomaly set; Among them, Represents the consistency deviation value of fields, represents the contextual semantic features of field k, represents the contextual semantic features of the comparison field k, represents the weight coefficient of field k in the semantic path, where p represents the total number of fields in the field group; Calculate the consistency deviation between fields to ensure that the same fields in different data sources maintain semantic consistency. Taking financial risk management as an example, the data of the same user's credit score and income are stored in different banks. However, due to different data collection methods, there are certain deviations. It is necessary to quantitatively calculate the consistency degree of fields and determine whether there is an overlap in the semantic paths of fields. This can be achieved through correlation analysis between fields. For example, in two data sources, the applicant's monthly income field is recorded as "monthly_income" and "income_per_month" by different institutions, but essentially represents the same content. By calculating the text similarity, it can be determined that they belong to the same field; For example, in loan approval, assume that the field group includes monthly income, debt ratio, and credit score, and the weights of these three fields are 、 、 , and the record of a certain applicant in Bank A is: monthly income 、debt ratio 、credit score , while the record in Bank B is: monthly income 、debt ratio 、credit score . Substitute the values into the formula for calculation: Calculate the numerator: ; Calculate the denominator: ; Calculate the consistency deviation value of the field:

[0030] The value obtained through calculation indicates that there are slight deviations in the field data of this user in the two data sources, but overall, they maintain a high degree of consistency. If the preset field consistency threshold is , then the deviation value of this field pair does not exceed the set range, so it will not be marked as abnormal. If the calculated value exceeds the set threshold, it is necessary to further analyze whether the field requires data cleaning or supplementing the data source. Based on the calculation results of all fields, screen the field pairs that exceed the threshold to obtain the set of field consistency anomalies. This set contains all fields with large differences between different data sources, providing a basis for subsequent data cleaning, field merging, or data standardization.

[0031] S313: Based on the set of field consistency exceptions, call the marked fields to determine the semantic path deviation situation, filter out the fields beyond the judgment range, perform consistency conflict marking, and obtain the metadata consistency conflict rate. Marking and processing the fields that exceed the consistency deviation threshold is widely applied in the actual enterprise resource planning (ERP). For example, in supply chain management, the consistency of product data and supplier data directly affects the efficiency of inventory control and order processing. The specific operations include calling the calculated set of field consistency exceptions, analyzing the deviation situation of each field pair. For instance, the inconsistency between the product number and the supplier number may lead to misdelivery or delay of orders. For the abnormal field pairs, consistency conflict marking is performed, which is completed by setting specific marking rules, such as setting the marking threshold for abnormal field pairs. This threshold is set based on past order error rates and customer complaint data. Through the calculation of actual data, for example, analyzing the order problems caused by the inconsistency between product data and supplier data in the past month, counting their occurrence frequency and impact degree, setting the threshold according to the data, and finally through analysis and marking, obtaining the metadata consistency conflict rate, which can intuitively point out the problem areas that need to be prioritized in data management, thereby guiding the data maintenance work of ERP, ensuring data consistency and accuracy, and improving the response speed and accuracy of the entire supply chain.

[0032] Please refer to Figure 5 , and the specific steps for obtaining the semantic path adjustment data are as follows: S411: Based on the metadata consistency conflict rate, extract the metadata change log of the data lake, identify the occurrence frequency of the alternative field path and the corresponding original path, combine the occurrence times of the original path fields, analyze the alternative frequency in the change log, filter out the paths with an alternative frequency higher than the set threshold, and obtain the data of the alternative high-frequency paths. Extract the occurrence frequency of the alternative field path and its corresponding original path based on the metadata change log of the data lake, and perform correlation analysis on the path data and the corresponding original path data. For example, in actual data management, an enterprise wants to analyze the most frequently updated data field paths in its data warehouse. By monitoring the daily generated metadata change log, it can identify the change frequency of the fields in a specific data table and calculate the occurrence frequency of each field in the log through an algorithm. Through the analysis process, not only can the structure adjustment of the data warehouse be optimized, but also the data model can be adjusted more precisely to improve data processing efficiency, thereby obtaining the data of the alternative high-frequency paths.

[0033] S412: Based on the data of the alternative high-frequency paths, analyze the alternative position adjustment degree of each path in the metadata of the data lake, judge the adaptability and position of the alternative path in the structure, and use the formula: ; Obtain structure position fitness data; Among them, represents the structure position fitness data, represents the path substitution frequency, represents the path appearance frequency in the original path, represents the total number of records in the change log, represents the log number when the alternative path first appears in the change log, represents the log number when the original path first appears in the change log, represents the total number of paths; It is necessary to calculate the substitution position adjustment degree of each path in the data lake metadata to judge its adaptability, and set the data of the substitution field path. For example, in the data lake management of an enterprise, a certain field path Due to architecture adjustment, it is replaced by multiple new paths, and the original path still exists in the data change log. By counting the change log records in the past month, the substitution frequency of the alternative path is 30 times, while the appearance frequency of the original path is 45 times, the total number of log records is counted as 1000, the alternative path first appears in log number 200, and the original path first appears in log number 150. Through the data; Substitute the specific values into the formula for calculation: ; This calculation shows that the structure position fitness of this path is 0.035, which means that the structure position adaptability of this alternative path in the data lake is relatively low. Therefore, in the subsequent adjustment process, more careful path replacement selection is required. If the path fitness is higher than a certain set benchmark value, such as 0.05, it can be considered that the adaptability of this path is good and the structure can be directly adjusted. Otherwise, the replacement plan needs to be further evaluated to obtain the structure position fitness data. ; S413: According to the structure position fitness data, screen the field paths in the alternative paths and replace the corresponding field positions of the original path to obtain semantic path adjustment data;

[0034] S413: According to the structure position fitness data, screen the field paths in the alternative paths and replace the corresponding field positions of the original path to obtain semantic path adjustment data; According to the structural position fitness data, screen those alternative paths with high fitness. Take practical operation examples. For instance, in a complex data set, a certain path shows a high structural position fitness due to frequent data changes. The administrator can select the path to replace the field positions of the original path. During this process, it involves readjusting the data model. By real-time monitoring the changes in performance, evaluate the data processing efficiency before and after path replacement. In practical applications, it involves regular performance evaluation reports so that the administrator can make the most appropriate decisions. This path selection and replacement strategy based on performance evaluation can effectively optimize the data processing flow, thereby obtaining semantic path adjustment data.

[0035] Please refer to Figure 6 , the specific steps for obtaining the metadata governance fusion structure are as follows: S511: Based on the semantic path adjustment data, extract the fields in the currently collected data, identify the word vector values of each field, analyze the semantic relationships of the fields in different data tables, and obtain the field word vector data; Use advanced data mining techniques to extract fields from the currently collected data, and calculate the word vector values of each field through a deep learning model. The process includes preprocessing of field data, model training, and vector generation. For example, in an e-commerce data lake, fields such as "user reviews" and "product descriptions" are extracted and converted into specific vector values through a word embedding model. The vector values can reflect the potential semantic correlations between fields. For example, during the training process of the Word2Vec model, the context environment of the fields is considered, and by adjusting model parameters such as window size and vector dimension, the accuracy and practicality of each field vector calculation are ensured. Through this method, the field vectors can be dynamically adjusted according to user behavior and product changes in different shopping seasons. Finally, the field word vector data is obtained, which not only provides similarity information between fields but also provides a scientific basis for the next data structure adjustment.

[0036] S512: Invoke the field word vector data, compare the fusion similarity of the field word vectors before and after adjustment, analyze the fusion matching degree of the fields after adjustment with the original structure, identify the grouping retention rate, and use the formula: ; Calculate the field fusion similarity data, analyze the similarity data, screen the fields that meet the grouping threshold, and obtain the field grouping retention rate data; Among them, represents the field fusion similarity data, and respectively represent the field word vector values before and after adjustment, represents the hierarchical depth of the field in the data table, represents the number of references of the field in the data lake, Represents the number of structural changes to the field; Compare the fusion similarity of the field word vectors before and after adjustment, and calculate the field grouping retention rate. Assume that in a medical data lake, two fields, "Patient Diagnosis" and "Treatment Record", have undergone data structure adjustments. The word vector values before the adjustment are 0.85, and the word vector values after the adjustment are 0.88. For the "Treatment Record", the word vector value before adjustment is 0.55, and after adjustment is 0.58. The hierarchical depth of these two fields in the database is 3, and the citation count is statistically 150 times, and the number of structural changes is statistically 2 times. Substitute the values into the formula for calculation: ; This calculation shows that the field "Patient Diagnosis" has a relatively high fusion similarity before and after adjustment, while the field "Treatment Record" has a relatively low fusion similarity before and after adjustment. Therefore, in the field grouping strategy, the "Patient Diagnosis" field needs to be merged, and the "Treatment Record" field needs further adjustment. The obtained field fusion similarity data will directly affect the metadata governance strategy in the data lake, ensure that the optimization of the data structure meets the business requirements, and provide a basis for subsequent field grouping.

[0037] S513: Based on the field grouping retention rate data, screen the eligible fields and merge them into a unified structure group to obtain the metadata governance fusion structure; Screen out the eligible fields and merge them into a unified structure group. For example, in a data lake in the financial industry, fields "Account Balance" and "Transaction Amount" show a high degree of fusion similarity and appropriate grouping retention rate, and will be merged into the same data model to optimize data query and analysis efficiency. This merging strategy is determined through a series of precise data analyses, including measurement of the correlation between fields and evaluation of grouping efficiency, to ensure that each merged field group can provide the maximum data integration benefit in the system. The final merging result will form the metadata governance fusion structure, which not only improves the efficiency of data processing but also reduces the complexity of data management.

[0038] The above is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any person skilled in the relevant art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A data lake metadata governance method based on semantic synthesis and text vectorization, characterized in that: The following steps are involved: S1: Based on the field name, data type and description information of the data source in the data lake, perform vector encoding of field content keywords and adjacent context words, extract the semantic co-occurrence group of the field in the differentiated context, identify the distance between semantic vectors, determine whether it is within the grouping range of similar fields, and obtain the field semantic aggregation amount; S2: based on the field semantic aggregation amount, extract the semantic center vector of the field belonging group, compare the cosine angle values ​​of the semantic center vectors between the field groups, determine whether the angle value is higher than the field semantic level, and obtain the field semantic hierarchical structure; S3: calling the field semantic hierarchical structure, extracting the context semantic structure, semantic path and semantic dependency relationship between fields of each group in the differentiated data source, judging whether it exceeds the set consistency judgment range, and if so, marking the field as a conflict item, and obtaining the metadata consistency conflict rate; S4: Based on the metadata consistency conflict rate, collect the replacement field paths in the data lake metadata change log, extract the frequency of occurrence of the replaced fields in the original path, screen the replacement high-frequency paths for structural position replacement, and obtain semantic path adjustment data.

2. The data lake metadata governance method based on semantic synthesis and text vectorization according to claim 1 is characterized in that: The field semantic collection quantity includes field attribution labels, contextual semantic features, and vector aggregation indicators. The field semantic hierarchical structure includes hierarchical grouping identifiers, semantic layer mapping, and field group hierarchical divisions. The metadata consistency conflict rate includes the conflict field ratio, field conflict distribution, and semantic conflict level. The semantic path adjustment data includes field replacement paths, structure adjustment records, and replacement frequency indicators.

3. The data lake metadata governance method based on semantic synthesis and text vectorization according to claim 1 is characterized in that: The specific steps for obtaining the field semantic collection amount are: S111: Based on the field name, data type and description information of the data source in the data lake, extract the keywords and context in the field, and analyze the word frequency and combination structure through vectorized coding to obtain the local semantic vector of the field; S112: calling the local semantic vector of the field, identifying the Euclidean distance and cosine similarity according to the semantic vector of the field in the differentiated context, and performing distance threshold judgment for the same type of fields, using the formula: ； Calculate the field semantic matching value, compare the matching value with the preset semantic attribution reference range, group and merge the field vectors that meet the reference, and obtain the field semantic attribution grouping quantity; in, Represents the semantic matching value of the field. Representative Field With Field The local semantic vector of Representative Field With Field Semantic energy value in local semantics, For Field With Field The cosine similarity value between the semantic vectors; S113: Based on the field semantic attribution grouping quantity, for fields that have not yet been attributed, determine whether the semantic matching values ​​with the fields in the attribution grouping meet the attribution conditions, retain the fields that do not meet the attribution conditions as independent fields, and obtain the field semantic collection quantity.

4. The data lake metadata governance method based on semantic synthesis and text vectorization according to claim 3 is characterized in that: The steps for obtaining the field semantic hierarchical structure are specifically as follows: S211: based on the field semantics collection amount, extract the semantic center vector of each group of fields, calculate the cosine angle value of the semantic center vectors between the field groups, select the field groups whose cosine angle value is lower than the preset field semantic level threshold, and obtain the preliminary semantic similarity between the field groups; S212: Call the preliminary semantic similarity between the field groups to determine whether it is higher than the field semantic level division standard value, identify the field groups that meet the standard, and merge them into the same level group using the formula: ； Get the semantic level merged value of the field; in, Represents the merged value of the field semantic level. Representative Field In the The semantic center component of the dimensional vector, Representative Field In the The semantic center component of the dimensional vector, Represents the number of vector dimensions; S213: calling the field semantic level merged value, arranging the merged field level group, analyzing the level structure mapping relationship, and obtaining the field semantic hierarchical structure.

5. The data lake metadata governance method based on semantic synthesis and text vectorization according to claim 4 is characterized in that: The steps for obtaining the metadata consistency conflict rate are specifically as follows: S311: calling the semantic level field group defined in the field semantic hierarchical structure, extracting the contextual semantic structure of the field in the differentiated data source, parsing the semantic path, and obtaining the field semantic dependency matrix; S312: According to the field semantic dependency matrix, the overlap of the field semantic paths is determined, and the field semantic consistency is identified in combination with the field context semantic structure, using the formula: ； Calculate the field consistency deviation value, filter out the field pairs that exceed the set range, and obtain the field consistency anomaly set; in, Represents the field consistency deviation value, represents the contextual semantic features of field k, represents the contextual semantic features of the comparison field k, represents the weight coefficient of field k in the semantic path, and p represents the total number of fields in the field group; S313: Based on the field consistency exception set, call the marked fields, determine the semantic path deviation, filter the fields beyond the judgment range, perform consistency conflict marking, and obtain the metadata consistency conflict rate.

6. The data lake metadata governance method based on semantic synthesis and text vectorization according to claim 5 is characterized in that: The steps for obtaining the semantic path adjustment data are specifically as follows: S411: based on the metadata consistency conflict rate, extract the data lake metadata change log, identify the replacement field path and the corresponding original path occurrence frequency, combine the number of occurrences of the original path field, analyze the replacement frequency in the change log, filter the path with a replacement frequency higher than a set threshold, and obtain the replacement high frequency path data; S412: Based on the alternative high-frequency path data, analyze the degree of adjustment of the alternative position of each path in the data lake metadata, determine the adaptability and position of the alternative path in the structure, and use the formula: ； Get structure position fitness data; in, Represents the structural position fitness data, Representative path The replacement frequency, Representative path The frequency of occurrence in the original path, Represents the total number of records in the change log, The log number representing the first occurrence of the alternative path in the change log, The log number representing the first occurrence of the original path in the change log. Indicates the total number of paths; S413: According to the structural position fitness data, the field path in the replacement path is screened, and the corresponding field position of the original path is replaced to obtain semantic path adjustment data.

7. The data lake metadata governance method based on semantic synthesis and text vectorization according to claim 1 is characterized in that: The method further comprises step S5: S5: Based on the semantic path adjustment data, extract the word vector value of the field in the current collected data, compare the fusion similarity and grouping retention rate of the field word vectors before and after the adjustment, merge the qualified fields into a unified structure group, and obtain the metadata governance fusion structure; The metadata governance fusion structure includes a fusion field group, a unified semantic structure, and a word vector similarity matrix.

8. The data lake metadata governance method based on semantic synthesis and text vectorization according to claim 7 is characterized in that: The steps for obtaining the metadata governance fusion structure are specifically as follows: S511: Based on the semantic path adjustment data, extract the fields in the current collected data, identify the word vector value of each field, analyze the semantic relationship of the fields in the differential data table, and obtain the field word vector data; S512: calling the field word vector data, comparing the fusion similarity of the field word vectors before and after the adjustment, analyzing the degree of fusion matching between the adjusted field and the original structure, identifying the grouping retention rate, and using the formula: ； Calculate the field fusion similarity data, analyze the similarity data, filter the fields that meet the grouping threshold, and obtain the field grouping retention rate data; in, Represents the field fusion similarity data, and Respectively represent the field word vector values ​​before and after adjustment, Represents the hierarchical depth of the field in the data table. Represents the number of times the field is referenced in the data lake. The number of times the structure of the representative field has changed; S513: Based on the field grouping retention rate data, filter out qualified fields, merge them into a unified structure group, and obtain a metadata governance fusion structure.

Images