Term extraction method based on library table metadata information

Abstract

The application discloses a term extraction method based on library table metadata information and relates to the technical field of data processing; in order to solve the technical problem of low efficiency and high cost caused by indiscriminate term extraction based on simple rules, the application solves the technical problem through the following steps: step one, data acquisition and storage; step two, data classification; step three, data importance quantification; and step four, differentiated term extraction. Through the combination of the three mechanisms of intelligent classification, accurate quantification and hierarchical management, the application successfully converts the indiscriminate extensive management into differentiated refined management, realizes efficient management by dividing, thereby concentrates limited manpower on core data, maximizes the management value, maintains the availability and retrievability of marginal and basic data, and improves the efficiency and accuracy of term extraction as a whole.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for extracting terms based on database table metadata information. Background Technology

[0002] The amount of database table metadata is huge and the sources are diverse. In this case, it is difficult to quickly understand and locate the database table metadata directly. This leads to problems such as difficulty in retrieving database table metadata, semantic confusion, frequent duplicate or conflicting names, and unclear field meanings. Therefore, it is very necessary to extract terms from database table metadata.

[0003] Current term extraction of database table metadata is usually based on simple rules (such as delimiter splitting, stop word filtering, etc.). This indiscriminate splitting and extraction of all metadata results in the extraction of a large number of low-value and meaningless terms. These terms need to be screened and cleaned up manually in the later stages, which is costly and inefficient. Summary of the Invention

[0004] Therefore, it is necessary to provide a term extraction method based on database table metadata information to address the problems mentioned in the background technology.

[0005] The objective of this invention can be achieved through the following technical solution: a term extraction method based on database table metadata information, comprising the following steps:

[0006] Step 1: Obtain and store the metadata of all database tables;

[0007] Step 2: Establish several databases of different types, and classify and process the metadata of the global database tables to allocate them to the corresponding database types;

[0008] Step 3: Identify the importance of each data entry in the database to output the sensitivity of each data entry;

[0009] Step four: Based on the sensitivity of data bars in the database, the data bars are packaged in batches to form edge data packets, basic data packets, and core data packets, and the corresponding term extraction methods are applied to different types of data packets.

[0010] In some embodiments, global database table metadata is categorized and assigned to databases of corresponding types:

[0011] Step 201: Split the metadata of the entire database table into several data bars according to the field granularity. Pre-set several different types of databases, and pre-set several feature words for each database. Use the feature words as anchors to locate each type of database.

[0012] Step 202: Clean and standardize the original text of the data bar and convert it into word token form. Use the TF-IDF model to identify and calculate the word tokens and keywords in the database to obtain the importance scores corresponding to the word tokens and keywords, and construct the word token vector of the data bar and the keyword vector of the database accordingly.

[0013] Step 203: Randomly select a database and its corresponding keyword vector, analyze the correlation between the keyword vector and the word vector to output the correlation degree. From this, the correlation degree between the data bar and all databases can be obtained. Select the database with the highest correlation degree as the target database of the data bar, and allocate and store the data bar in the target database. In this way, all data bars can be allocated and stored in the database with the highest correlation degree.

[0014] In some embodiments, the word vectors of the data bars and the keyword vectors of the database are constructed:

[0015] The TF-IDF model is used to identify and calculate the importance score of each term. Each data point is represented as a term vector, where each dimension of the term vector corresponds to a term, and its value is the importance score of that term. The term vector is represented as follows: m1 represents the total number of lexical units;

[0016] Similarly, by merging the feature words from all databases to obtain a feature word library, the TF-IDF model is used to identify and calculate the importance scores for all feature words in any given database. This process is repeated across all databases to obtain the importance score for each feature word in each database. A keyword vector is then constructed, where each dimension represents a keyword, and its value is the importance score corresponding to that keyword. The keyword vector is represented as follows: m2 represents the total number of keywords.

[0017] In some embodiments, the correlation between keyword vectors and word vectors is analyzed:

[0018] The NLP model is used to compare the word vectors of the data entries with the keyword vectors, filter out the shared words that overlap between the two, extract the importance scores of the shared words in the database and in the data entries, and construct the shared vector denoted as . Where n is the total number of shared words, k∈n, and k represents the index of any shared word; C n G represents the importance score of the nth shared word in the data bar. n The value of the nth shared word in the database is represented by its importance score. The cosine similarity formula is used to evaluate the semantic relevance between the data entry and the database, yielding the correlation score. The calculation formula is as follows:

[0019] .

[0020] In some embodiments, the importance of each data bar in the database is identified to output the sensitivity of each data bar:

[0021] Step 301: Pre-set each keyword in the feature word library to correspond to an importance value. Compare all the lexical units contained in each data bar with all the feature words in the feature word library to match the importance value corresponding to each lexical unit. If the match fails, the importance value corresponding to the lexical unit is zero. Iterate through the importance values ​​of all lexical units and take the lexical units with importance values ​​> 0 as the valid lexical units of the data bar. Analyze the business keyness of the data bar based on the valid lexical units of the data bar and their corresponding importance values ​​to output the business value.

[0022] Step 302: Obtain the access parameters of the data bar. The specific access parameters include the number of accesses and the access time for each access. Perform access activity analysis based on the access parameters to output the access value.

[0023] Step 303: Obtain the reference parameters of the data bar, including the number of references and the reference time of each reference; perform structural dependency analysis based on the reference parameters to output the access value;

[0024] Step 304, access value H 访问 and reference value H 引用 Univariate scaling yields the compressed access value 'a' and reference value 'b'. The specific compression formula is as follows:

[0025]

[0026] The compressed access value 'a', reference value 'b', and business value 'Y' are then used to perform formulaic calculations and analyses to obtain the sensitivity of the data bars. From this, the sensitivity of all data bars in the database can be obtained. The specific calculation formula is as follows:

[0027] .

[0028] In some embodiments, the business criticality of data bars is analyzed to output business values:

[0029] Extract the effective words and their corresponding importance values ​​from each data entry in the database. Select the largest importance value and divide it by the sum of the importance values ​​of the effective words to obtain the concentration ratio P1. At the same time, calculate the number of effective words and divide it by the total number of words in the data entry to obtain the effective ratio, denoted as P2.

[0030] The business value is calculated by using the linear weighted fusion calculation formula Y=α×P1+β×P2 to obtain the concentration ratio P1 and the effective ratio P2, where α and β are the weight factors of the concentration ratio and the effective ratio, respectively, and α+β=1. The technical personnel of this institution have set the values ​​of α and β to 0.7 and 0.3, respectively.

[0031] In some embodiments, access activity analysis is performed based on access parameters and structural dependency analysis is performed based on reference parameters:

[0032] Access activity analysis based on access parameters:

[0033] The access scale factor is obtained by compressing and smoothing the number of accesses using a logarithmic function. The logarithmic function is:

[0034]

[0035] Where N represents the total number of visits;

[0036] The time difference between each access time and the current time is calculated to obtain the time distance of each access, denoted as Δt. i Where i is a positive integer, i represents the index of any access; then all access counts are sorted according to their corresponding time distance Δt. i The access time factor is obtained by applying time decay. The time decay formula is:

[0037]

[0038] Where λ 访问 To control the decay rate, the access decay rate is controlled.

[0039] Finally, the access scale factor is multiplied by the access time factor to obtain the access value corresponding to the data bar.

[0040] Similarly, structural dependency analysis is performed based on reference parameters: the reference parameters of the data bar are obtained, including the number of references and the reference time of each reference. The reference scale factor is obtained by compressing and smoothing the reference number using a logarithmic function. The time distance of each reference is calculated by the time difference between the reference time corresponding to each reference and the current time. Then, all reference counts are time-decayed according to their corresponding time distances to obtain the reference elapsed factor. Finally, the reference scale factor is multiplied by the reference elapsed factor to obtain the reference value corresponding to the data bar.

[0041] In some embodiments, data bars are packaged in batches based on their sensitivity within the database:

[0042] A preset sensitivity range is defined. The sensitivity of each data bar is compared and analyzed with the sensitivity range. If the sensitivity is less than the lower limit of the sensitivity range, the data bar is classified as edge data. If the sensitivity is within the sensitivity range, the data bar is classified as basic data. If the sensitivity is greater than the upper limit of the sensitivity range, the data bar is classified as core data. All data bars in the database are summarized, packaged, and output as edge data packets, basic data packets, and core data packets according to their corresponding data types.

[0043] In some embodiments, corresponding term extraction methods are performed for different types of data packets:

[0044] Extract the word vectors of each data bar in the edge data packet, use all its words as search tags, and build an index that can be retrieved by the global search engine based on all search tags;

[0045] Extract the basic data package, perform automated term extraction on it to form the final terminology, and output it.

[0046] Extract the core data packet, and use the word vectors of each data item in the core data packet as terminology auxiliary data. Send the core data packet to the corresponding responsible person for terminology extraction according to the database type of the core data packet.

[0047] In some embodiments, automated term extraction forms the final terms:

[0048] Extract the word vectors of each data item in the basic data package. m1 represents the total number of lexical units. A preset score threshold is set. The importance score corresponding to each lexical unit vector is compared and analyzed with the score threshold. If the importance score is greater than or equal to the score threshold, the lexical unit corresponding to the importance score is used as the defined term of the data bar. The number of defined terms in the data bar is counted. When the number of defined terms is greater than 0, the defined terms are directly output as the final terms of the data bar. When the number of defined terms is less than 0, all lexical units in the data bar are sorted according to their corresponding importance scores. The top 3 lexical units are selected as the supplementary terms of the data bar and output as the final terms of the data bar.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1. This invention automatically classifies the messy global database metadata into different thematic databases by using a preset business feature lexicon and TF-IDF vectorization, thereby achieving the initial ordering of data assets and business semantic association. In this process, the semantic relevance between data bars and the database is measured by cosine similarity, making the classification more objective and accurate, laying the foundation for subsequent divide-and-conquer, so that data can be aggregated according to business domains.

[0051] 2. This invention achieves precise quantification of the sensitivity of each data point through the joint calculation of business value, access value, and citation value. The business value calculation process takes into account both the concentration ratio and the effective ratio, and is dominated by effective keywords, so that the results can better reflect the business criticality of the data. Logarithmic compression and time decay are introduced into the access value and citation value respectively to capture the overall popularity of long-term access, while highlighting the timeliness of access and citation, so as to quickly identify hot spots and key data with high dependence. Finally, the three are integrated through a non-linear formula, making the sensitivity smoother and more robust, avoiding misjudgments caused by the anomaly of a single indicator.

[0052] 3. Based on the sensitivity calculation results, the data bars in the database are divided into three categories: edge, basic, and core. Edge data packets, basic data packets, and core data packets are generated. Differentiated term extraction strategies are adopted for different types of data packets to achieve efficient governance through divide and conquer. This allows limited manpower to be concentrated on core data to maximize governance value, while maintaining the availability and searchability of edge and basic data, thereby improving the overall efficiency and accuracy of term extraction. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Current term extraction methods for database table metadata typically rely on simple rules (such as delimiter splitting and stop word filtering) for splitting and extraction. This indiscriminate splitting and extraction of all metadata results in the extraction of a massive amount of low-value and meaningless terms. These terms require extensive manual screening and cleanup, which is costly and inefficient. To address this technical problem, this solution combines data grading with a differentiated term extraction strategy, fundamentally solving the inefficiency and resource waste caused by traditional indiscriminate processing methods.

[0057] As Figure 1 shown, the term extraction method based on library table metadata information includes the following steps:

[0058] Step 1, obtain the global library table metadata and store it;

[0059] Step 2, establish several databases of different types, and classify the global library table metadata for distribution to the corresponding type of database; specifically:

[0060] Step 201, split the global library table metadata into a large number of data entries according to the field granularity. Presuppose that there are several databases of different types. Those skilled in the art preset several feature words for each database according to the actual needs of the application (the business type of the enterprise) to use the feature words as anchors to locate each type of database; it should be noted that presetting feature words for each database is set by those skilled in the art based on multi-dimensional mining of the technical mode, business knowledge, data content and enterprise standards of this application scenario;

[0061] Step 202, clean and standardize the original text of the data entry into the form of word tokens. The specific cleaning and standardization process is as follows: split the original text of the data entry into independent words, and convert all the letters in it into lowercase to ensure case insensitivity, and then filter out the meaningless words to obtain several word tokens corresponding to the data entry. A word token is the basic unit in lexicology; the meaningless words specifically include: "a", "this", "that", "le", "de", etc.; use the TF-IDF (term frequency-inverse document frequency) model to identify and calculate the word tokens of the data entry to obtain the important score corresponding to the word token. It should be noted that the important score refers to the importance of the word token in the current data entry, and at the same time combines its distribution universality in all data entries; for example, a certain word token may appear in thousands of data entries, so its IDF value will be very low. Even if its TF value is high in a certain data entry, its TF-IDF value (important score) will be pulled down; on the contrary, a certain word token may only appear in a few financial-related data entries, and its IDF value will be very high; where the important score of the TF-IDF (term frequency-inverse document frequency) model = TF×IDF; the higher the important score, the more likely the word token is to become the key feature representing the uniqueness of this data entry; thus, each data entry can be represented as a word token vector. Each dimension of the word token vector corresponds to a word token, and its value is the important score of the word token. The word token vector is represented as , where m1 represents the total number of word tokens;

[0062] Similarly, the feature words corresponding to all databases are merged to obtain a feature word library. Simultaneously, the TF-IDF (Term Frequency-Inverse Document Frequency) model is used to identify and calculate the importance scores for all feature words in any given database. This process is repeated across all databases to obtain the importance score for each feature word in each database. A keyword vector is then constructed, where each dimension represents a keyword, and its value is the importance score corresponding to that keyword. The keyword vector is represented as follows: m2 represents the total number of keywords;

[0063] Step 203: Randomly select a database and its corresponding keyword vector. Use an NLP (Natural Language Processing) model to compare the word vectors of the data entries with the keyword vectors, and filter out shared words that overlap between the two (shared words refer to words that appear simultaneously in both the data entries and the database). This indicates that the shared word appears in both the data entries and the database. Extract the importance scores of the shared word in the database and in the data entries, and construct a shared vector denoted as . Where n is the total number of shared words, k∈n, k represents the index of any shared word, n≤m1, n≤m2; C n G represents the importance score of the nth shared word in the data bar. n The value of the nth shared word in the database is represented by its importance score. The cosine similarity formula is used to evaluate the semantic relevance between the data entry and the database, yielding the correlation score. The calculation formula is as follows:

[0064]

[0065] The closer the correlation value is to 1, the more relevant the data bar is to the topic of the database; the closer the value is to 0, the less relevant it is. From this, we can obtain the correlation between the data bar and all databases. Select the database with the highest correlation as the target database of the data bar and allocate the data bar to the target database. In this way, we can allocate and store all data bars to the database with the highest correlation.

[0066] By pre-setting a business feature lexicon and using TF-IDF vectorization, the messy global database metadata is automatically classified into different thematic databases, achieving initial ordering of data assets and business semantic association. In this process, cosine similarity is used to measure the semantic relevance between data bars and the database, ensuring the objectivity and accuracy of classification, laying the foundation for subsequent divide-and-conquer, and enabling data to be aggregated according to business domains.

[0067] Step 3: Identify the importance of each data entry in the database to output the sensitivity of each data entry; specifically:

[0068] Step 301: Pre-set each keyword in the feature word library to correspond to an importance value. It should be noted that the importance value of the keyword is a quantitative weight assigned to each keyword by those skilled in the art based on the actual needs of the enterprise in the actual business process. It is used to accurately represent the relative importance level of the business concept represented by the keyword in the data governance system. Its design is based on a deep understanding of the core business processes, compliance requirements and data asset value. It transforms subjective business knowledge into objective and calculable metrics, which can guide the data governance process to accurately focus on high-value data assets and achieve differentiated management and optimal resource allocation for data of different importance.

[0069] Each data entry's constituent words are compared with all feature words in the feature word library to match their corresponding importance values. If a match fails, meaning there is no keyword in the feature word library corresponding to the constituent word, then the importance value of the constituent word is zero. The importance values ​​of all constituent words are iterated, and constituent words with importance values ​​> 0 are considered valid constituent words for each data entry. This yields the valid constituent words and their corresponding importance values ​​for each data entry in the database. The highest importance value is selected and divided by the sum of the importance values ​​of the valid constituent words to obtain the concentration ratio P1. Simultaneously, the number of valid constituent words is calculated and divided by the total number of constituent words in the data entry to obtain the effective ratio, denoted as P2. It should be noted that a larger concentration ratio indicates that the data entry depends on a major constituent word (i.e., a highly important business concept dominates). For example, if a data entry contains the constituent word "net profit" with a high importance value, while other constituent words have lower importance values, the concentration ratio is close to 1, indicating that the business importance of this data entry is higher and the information contained in the data entry is more sensitive. A larger effective ratio indicates that most constituent words in the data entry match keywords in the feature word library, suggesting that the data entry is highly relevant to the business theme.

[0070] The business value is calculated by using the linear weighted fusion formula Y=α×P1+β×P2 to calculate the concentration ratio P1 and the effective ratio P2, where α and β are the weighting factors of the concentration ratio and the effective ratio, respectively, and α+β=1. The technical personnel of this institution have set the values ​​of α and β to 0.7 and 0.3, respectively. This linear weighted fusion formula is based on the business principle that core is superior to broadness. By giving a higher weight to the concentration ratio, the final business value can keenly capture the data points that contain high-value and high-sensitivity business concepts.

[0071] Step 302: Obtain the access parameters for the data bars. Specific access parameters include the number of accesses and the access time for each access. It should be noted that access refers to the act of reading and querying during runtime. Some data bars may be accessed extremely frequently, while most data bars are accessed less frequently, leading to a large numerical range. Therefore, it is necessary to compress and smooth the access count. A logarithmic function is used to process the access count to obtain the access scale factor. The logarithmic function is:

[0072]

[0073] Where N represents the total number of accesses; the access scale factor represents the overall scale of data entries accessed, eliminating huge orders of magnitude differences between different data entries, making the results more stable and more comparable;

[0074] The time difference between each access time and the current time is calculated to obtain the time distance of each access, denoted as Δt. i Where i is a positive integer, i represents the index of any access; then all access counts are sorted according to their corresponding time distance Δt. i The access time factor is obtained by applying time decay. The time decay formula is:

[0075]

[0076] Where λ 访问 The access decay rate, controlling the decay speed, is set to a value of 0.02, λ. 访问 The larger the value, the faster the access record depreciates in value.

[0077] Finally, the access scale factor is multiplied by the access timeliness factor to obtain the access value corresponding to the data bar, denoted as H. 访问 The access value is obtained by integrating the historical access scale and recent access activity of the data. The logarithmic function compresses the total number of accesses to eliminate the influence of extreme values ​​and reflect the overall popularity. Then, the exponential time decay model highlights the higher weight of recent accesses to capture timeliness. Finally, the two are multiplied to obtain the access value. It can accurately identify continuously active data (high total number of accesses and frequent recent accesses), quickly discover emerging hot data (low total number of accesses but recent surge), and automatically downplay data that was once popular but has cooled down or has been low-frequency for a long time.

[0078] Step 303, similarly, obtain the reference parameters for the data bar. Specific reference parameters include the number of references and the reference time for each reference. It should be noted that a reference refers to the dependency or declaration behavior of downstream assets on the data bar at the design or operational level. The reference count is compressed and smoothed using a logarithmic function to obtain the reference scale factor. The logarithmic function is:

[0079]

[0080] Where M represents the total number of citations; the citation scale factor represents the overall scale of the data bar being cited, eliminating huge orders of magnitude differences between different data bars, making the results more stable and more comparable;

[0081] The time difference between each reference time and the current time is calculated to obtain the time distance of each reference, denoted as Δt. j Where j is a positive integer, j represents the index of any reference; then all reference counts are sorted according to their corresponding time distance Δt. j The time decay is used to obtain the reference aging factor. The time decay formula is:

[0082]

[0083] Where λ 引用 To control the decay rate, the value is set to 0.005, λ. 引用 The larger the value, the faster the reference record depreciates;

[0084] Finally, the citation size factor is multiplied by the citation timeliness factor to obtain the citation value corresponding to the data bar, denoted as H. 引用 ;

[0085] Step 304: After steps 301 to 304, the business value Y and access value H of all data bars can be obtained. 访问 and reference value H 引用 Then, the access value and reference value are univariately scaled to obtain the compressed access value 'a' and reference value 'b'. The specific compression formula is as follows:

[0086]

[0087] Next, the sensitivity S of the data bar is obtained by formulaically calculating and analyzing the business value Y, the compressed access value a, and the reference value b. From this, the sensitivity of all data bars in the database can be obtained; the specific calculation formula is as follows:

[0088]

[0089] This formula combines the business value, access value (access activity), and reference value (structural dependency) of a data bar through a product term, and then transforms it into a sensitivity between 0 and 1 through a monotonically convex mapping of 1-exp(...), avoiding the linear explosion of extreme values. The product structure means that the three complement each other, and high sensitivity is triggered only when the business value is high and there is strong access or reference support at the same time. If a certain item is extremely low (for example, the business semantics are extremely weak or the reference is 0), the sensitivity of the data bar is naturally suppressed, avoiding misjudgment due to a single signal.

[0090] The sensitivity of each data point is accurately quantified through the joint calculation of three indicators: business value, access value, and citation value. The calculation of business value takes into account both the concentration ratio and the effective ratio, and is dominated by effective keywords, so that the results can better reflect the business criticality of the data. Logarithmic compression and time decay are introduced for access value and citation value respectively to capture the overall popularity of long-term access, while highlighting the timeliness of access and citation, so as to quickly identify hot spots and key data with high dependence. Finally, the three are integrated through a non-linear formula to make the sensitivity smoother and more robust, and avoid misjudgment caused by the anomaly of a single indicator.

[0091] Step four involves batching data bars into edge data packets, basic data packets, and core data packets based on their sensitivity within the database. Corresponding terminology extraction methods are then applied to different types of data packets to achieve a divide-and-conquer approach to terminology extraction, improving both efficiency and accuracy. Specifically:

[0092] Step 401: Preset a sensitive interval. Those skilled in the art set the sensitivity interval to [0.3, 0.7]. The preset sensitive interval [0.3, 0.7] is determined by those skilled in the art based on business practice, data distribution characteristics, and governance cost-benefit analysis. The sensitivity calculation results of most data bars will follow a normal distribution or a concentrated distribution. The interval [0.3, 0.7] can effectively cover the main part of the distribution, avoiding classification imbalance due to excessively high or low thresholds (such as too much data being classified as core or edge). Compare and analyze the sensitivity of the data bar with the sensitive interval. If the sensitivity is less than the lower limit of the sensitive interval, the data bar is classified as edge data. If the sensitivity is within the sensitive interval, the data bar is classified as basic data. If the sensitivity is greater than the upper limit of the sensitive interval, the data bar is classified as core data. All data bars in the database are summarized and packaged according to their corresponding data types (edge ​​data, basic data, and core data) to output edge data packets, basic data packets, and core data packets. Edge data packets execute step 402, basic data packets execute step 403, and core data packets execute step 404.

[0093] Step 402: Extract the word vectors of each data entry in the edge data package, and use all words as search tags. Then, build an index that can be retrieved by the global search engine based on all search tags. Collect all words corresponding to each data entry in the edge data package and write them into the inverted index structure of the global search engine. This will enable the user to quickly locate the corresponding data entry by matching the index when the user enters any word or field fragment. In this way, even if the edge data is not included in the enterprise's official terminology database, it can still be globally searchable and quickly discovered, while greatly reducing the cost of manual governance and maintaining a lightweight data governance strategy.

[0094] Step 403: Extract the word vectors of each data bar in the basic data packet. m1 represents the total number of lexical units. A score threshold is preset. The score threshold is selected by analyzing the distribution of the importance scores of all lexical units in the basic data package (histogram) and selecting a certain quantile (e.g., 70th percentile) as the score threshold. The importance scores corresponding to each lexical unit vector are compared with the score threshold. If the importance score is greater than or equal to the score threshold, the lexical unit corresponding to the importance score is used as the defined term of the data bar. The number of defined terms in the data bar is counted. If the number of defined terms is greater than 0, the defined terms are directly output as the final terms of the data bar. If the number of defined terms is less than 0, all lexical units in the data bar are sorted according to their corresponding importance scores, and the top 3 lexical units are selected as the supplementary terms of the data bar. The supplementary terms are output as the final terms of the data bar. It should be noted that the defined data and supplementary terms of the data bar are distinguished and marked with different colors.

[0095] Step 404: Extract the core data package. Use the word vectors of each data entry in the core data package as auxiliary data for terminology extraction. Send the core data package to the corresponding responsible person for terminology extraction according to the database type of the core data package. It should be noted that the database type is set by those skilled in the art based on the business type of the enterprise. Different types of databases are pre-set to have corresponding responsible persons for the business type of terminology extraction. Distribute the word vectors of each data entry in the core data package as auxiliary data for terminology extraction to the responsible persons in the corresponding fields. Providing pre-processed word vectors with importance weights can save the time of understanding and screening from scratch, and change the workload from information mining to information confirmation and refinement, which can significantly improve the efficiency and accuracy of manual terminology extraction.

[0096] Based on the sensitivity calculation results, the data bars in the database are divided into three categories: edge, basic, and core. Edge data packets, basic data packets, and core data packets are generated. Differentiated term extraction strategies are adopted for different types of data packets to achieve efficient governance through divide and conquer. This allows limited manpower to be concentrated on core data to maximize governance value, while maintaining the availability and searchability of edge and basic data, thereby improving the overall efficiency and accuracy of term extraction.

[0097] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A term extraction method based on database table metadata information, characterized in that, Includes the following steps: Step 1: Obtain metadata for all database tables; Step 2: Establish several databases of different types, and classify and process the metadata of the global database tables to allocate them to the corresponding database types; Step 3: Identify the importance of each data entry in the database to output the sensitivity of each data entry; Step four: Based on the sensitivity of data bars in the database, the data bars are packaged in batches to form edge data packets, basic data packets, and core data packets, and the corresponding term extraction methods are applied to different types of data packets.

2. The term extraction method based on database table metadata information according to claim 1, characterized in that, The metadata of the entire database tables is categorized and assigned to the corresponding database types, including: Step 201: Split the metadata of the entire database table into several data bars according to the field granularity. Pre-set several different types of databases, and pre-set several feature words for each database. Use the feature words as anchors to locate each type of database. Step 202: Clean and standardize the original text of the data bar and convert it into word token form. Use the TF-IDF model to identify and calculate the word tokens and keywords in the database to obtain the importance scores corresponding to the word tokens and keywords, and construct the word token vector of the data bar and the keyword vector of the database accordingly. Step 203: Randomly select a database and its corresponding keyword vector, analyze the correlation between the keyword vector and the word vector to output the correlation degree. From this, the correlation degree between the data bar and all databases can be obtained. Select the database with the highest correlation degree as the target database of the data bar, and allocate and store the data bar in the target database. In this way, all data bars can be allocated and stored in the database with the highest correlation degree.

3. The term extraction method based on database table metadata information according to claim 2, characterized in that, Constructing the word vectors for the data bars and the keyword vectors for the database: The TF-IDF model is used to identify and calculate the importance score of each term. Each data point is represented as a term vector, where each dimension of the term vector corresponds to a term, and its value is the importance score of that term. The term vector is represented as follows: m1 represents the total number of lexical units; Similarly, by merging the feature words from all databases to obtain a feature word library, the TF-IDF model is used to identify and calculate the importance scores for all feature words in any given database. This process is repeated across all databases to obtain the importance score for each feature word in each database. A keyword vector is then constructed, where each dimension represents a keyword, and its value is the importance score corresponding to that keyword. The keyword vector is represented as follows: m2 represents the total number of keywords.

4. The term extraction method based on database table metadata information according to claim 3, characterized in that, Analyzing the relationship between keyword vectors and word vectors includes: The NLP model is used to compare the word vectors of the data entries with the keyword vectors, filter out the shared words that overlap between the two, extract the importance scores of the shared words in the database and in the data entries, and construct the shared vector denoted as . Where n is the total number of shared words, k∈n, and k represents the index of any shared word; C n G represents the importance score of the nth shared word in the data bar. n The value of the nth shared word in the database is represented by its importance score. The cosine similarity formula is used to evaluate the semantic relevance between the data entry and the database, yielding the correlation score. The calculation formula is as follows: 。 5. The term extraction method based on database table metadata information according to claim 4, characterized in that, Identify the importance of each data row in the database to output the sensitivity of each data row, including: Step 301: Pre-set each keyword in the feature word library to correspond to an importance value. Compare all the lexical units contained in each data bar with all the feature words in the feature word library to match the importance value corresponding to each lexical unit. If the match fails, the importance value corresponding to the lexical unit is zero. Iterate through the importance values ​​of all lexical units and take the lexical units with importance values ​​> 0 as the valid lexical units of the data bar. Analyze the business keyness of the data bar based on the valid lexical units of the data bar and their corresponding importance values ​​to output the business value. Step 302: Obtain the access parameters of the data bar. The specific access parameters include the number of accesses and the access time for each access. Perform access activity analysis based on the access parameters to output the access value. Step 303: Obtain the reference parameters of the data bar, including the number of references and the reference time of each reference; perform structural dependency analysis based on the reference parameters to output the access value; Step 304: Perform univariate scaling on the access value and reference value to obtain the compressed access value and reference value, and then perform formulaic calculation and analysis with the business value to obtain the sensitivity of the data bar. From this, the sensitivity of all data bars in the database can be obtained.

6. The term extraction method based on database table metadata information according to claim 5, characterized in that, Analyze the business criticality of data bars to output business values, including: Extract the effective words and their corresponding importance values ​​from each data entry in the database. Select the largest importance value and divide it by the sum of the importance values ​​of the effective words to obtain the concentration ratio. At the same time, calculate the number of effective words and divide it by the total number of words in the data entry to obtain the effective ratio. The business value is obtained by linearly weighting and merging the concentrated proportion and the effective proportion.

7. The term extraction method based on database table metadata information according to claim 6, characterized in that, Access activity analysis based on access parameters and structural dependency analysis based on reference parameters include: Access activity analysis based on access parameters: The access scale factor is obtained by compressing and smoothing the number of accesses using a logarithmic function. The logarithmic function is: Where N represents the total number of visits; The time difference between each access time and the current time is calculated to obtain the time distance of each access, denoted as Δt. i Where i is a positive integer, i represents the index of any access; then all access counts are sorted according to their corresponding time distance Δt. i The access time factor is obtained by applying time decay. The time decay formula is: Where λ 访问 To control the decay rate, the access decay rate is controlled. Finally, the access scale factor is multiplied by the access time factor to obtain the access value corresponding to the data bar. Similarly, structural dependency analysis is performed based on reference parameters: the reference parameters of the data bar are obtained, including the number of references and the reference time of each reference. The reference scale factor is obtained by compressing and smoothing the reference number using a logarithmic function. The time distance of each reference is calculated by the time difference between the reference time corresponding to each reference and the current time. Then, all reference counts are time-decayed according to their corresponding time distances to obtain the reference elapsed factor. Finally, the reference scale factor is multiplied by the reference elapsed factor to obtain the reference value corresponding to the data bar.

8. The term extraction method based on database table metadata information according to claim 7, characterized in that, Data bars are packaged in batches based on their sensitivity within the database, including: A preset sensitivity range is defined. The sensitivity of each data bar is compared and analyzed with the sensitivity range. If the sensitivity is less than the lower limit of the sensitivity range, the data bar is classified as edge data. If the sensitivity is within the sensitivity range, the data bar is classified as basic data. If the sensitivity is greater than the upper limit of the sensitivity range, the data bar is classified as core data. All data bars in the database are summarized, packaged, and output as edge data packets, basic data packets, and core data packets according to their corresponding data types.

9. The term extraction method based on database table metadata information according to claim 8, characterized in that, Perform corresponding term extraction methods for different types of data packets, including: Extract the word vectors of each data bar in the edge data packet, use all its words as search tags, and build an index that can be retrieved by the global search engine based on all search tags; Extract the basic data package, perform automated term extraction on it to form the final terminology, and output it. Extract the core data packet, and use the word vectors of each data item in the core data packet as terminology auxiliary data. Send the core data packet to the corresponding responsible person for terminology extraction according to the database type of the core data packet.

10. The term extraction method based on database table metadata information according to claim 9, characterized in that, Automated terminology extraction forms the final terminology, including: Extract the word vectors of each data item in the basic data package. m1 represents the total number of lexical units. A preset score threshold is set. The importance score corresponding to each lexical unit vector is compared and analyzed with the score threshold. If the importance score is greater than or equal to the score threshold, the lexical unit corresponding to the importance score is used as the defined term of the data bar. The number of defined terms in the data bar is counted. When the number of defined terms is greater than 0, the defined terms are directly output as the final terms of the data bar. When the number of defined terms is less than 0, all lexical units in the data bar are sorted according to their corresponding importance scores. The top 3 lexical units are selected as the supplementary terms of the data bar and output as the final terms of the data bar.

Images