Classified labeling method for database metadata based on artificial intelligence

Abstract

The invention discloses a database metadata classification labeling method based on artificial intelligence. According to the method, artificial intelligence is applied to the database metadata classification marking process, and multiple key steps such as data preparation and preprocessing, model training and optimization, classification marking process and follow-up treatment and application are involved. The steps jointly form a complete artificial intelligence database metadata classification marking process, and powerful support and guarantee are provided for database management and data governance.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for classifying and labeling database metadata based on artificial intelligence. Background Art

[0002] With the explosive growth of data volume, traditional database management systems face huge challenges in processing massive data, achieving efficient query and optimizing resource utilization.

[0003] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for classifying and labeling metadata in a database based on artificial intelligence, so as to automatically classify and label the metadata in the database, reduce manual intervention, and improve processing efficiency.

[0005] The technical solution of the present invention is as follows: a method for classifying and labeling database metadata based on artificial intelligence is provided, comprising the following steps.

[0006] S1: Data preparation and preprocessing.

[0007] Data collection: Extract metadata from the database. Metadata includes table name, field name, data type, description information, etc.; determine the scope and quantity of metadata that needs to be classified and labeled based on actual needs.

[0008] Data cleaning: Clean the extracted metadata to remove invalid, duplicate, incomplete and erroneous data; format the metadata to ensure data consistency and standardization.

[0009] Data annotation: Annotate the cleaned metadata to provide a training set for subsequent machine learning model training.

[0010] S2: Feature extraction.

[0011] Analyze the characteristics of metadata, such as data type, data format, data value range, etc., and extract features that are useful for classification.

[0012] Use natural language processing techniques (such as word segmentation, word frequency statistics, etc.) to extract features from text-type metadata.

[0013] S3: Model training and optimization.

[0014] S31: Select a suitable algorithm model, according to the characteristics of the metadata and classification requirements, select a suitable machine learning algorithm model; preferably, for the metadata name or description, select a basic text similarity algorithm model such as cosine similarity or Jaccard similarity; for the metadata content, select a BERT model suitable for text classification tasks, such as BERT-base-uncased or BERT-base-chinese (if the metadata is in Chinese). The algorithm model is a convolutional neural network model built using a deep learning framework such as Keras or TensorFlow.

[0015] S32: Training model; using the annotated metadata as a training set to train the machine learning model; improving the accuracy and generalization ability of the model by adjusting model parameters, optimizing algorithms, etc.

[0016] Taking the BERT model as an example, the specific method is: Download the model and weights: download the pre-trained BERT model and its weights from model libraries such as Hugging Face; Model fine-tuning: use the annotated metadata to fine-tune the BERT model to adapt to specific classification tasks; During the fine-tuning process, you need to set appropriate hyperparameters, such as learning rate, batch size, etc.

[0017] S33: Use the validation set to evaluate the trained BERT model. The evaluation indicators may include accuracy, recall, F1 score, etc. Based on the evaluation results, optimize and adjust the BERT model to improve the performance of the BERT model.

[0018] S4: Subsequent governance and application; including: data organization and management, data quality monitoring, data analysis and mining. Specifically:

[0019] S41: Data organization and management; Based on the results of classification and labeling, classify and organize the metadata in the database to improve the discoverability and availability of data; use label information to implement data permission control and access management to ensure data security and compliance.

[0020] S42: Data quality monitoring; regularly monitor and analyze the quality of metadata in the database to promptly identify and resolve data quality issues; apply the corresponding annotated metadata (data type, format, value range, etc.) according to the annotated data category, conduct quality inspections and analyze the results; evaluate the changing trends of data quality by comparing labeled data at different time points, and provide decision support for data governance; evaluate data quality improvements based on the proportion of data that do not meet metadata specification requirements at different time points.

[0021] S43: Data analysis and mining: Use the metadata after classification and labeling to conduct more in-depth data analysis and mining to discover the association rules and potential value in the data; understand the association relationship between various business models in the application system that lacks or is missing entity relationships. Break the data silos, cross organizations and systems, integrate different business data, unify data models, and improve data quality.

[0022] The annotation process involves the classification, description, and attribute extraction of metadata;

[0023] Metadata classification is divided into three categories: business metadata (mainly describes the concepts, rules and processes of the business, such as business terms, business rules, data models, etc. It helps business personnel better understand the application and significance of data in the business), technical metadata (describes information such as the physical storage, technical architecture and data processing process of the data, such as database table structure, field type, data storage path, etc. It is crucial for technical personnel to manage and maintain data), and management metadata (information related to data lifecycle management, data quality, data security, etc. It helps managers monitor and manage the entire life cycle of data to ensure the accuracy, integrity and security of data).

[0024] Metadata description refers to the accurate definition of the object represented by the metadata. It cannot be without content and cannot be ambiguous.

[0025] Metadata attributes are the information storage method of metadata: such as data type, data length, precision, format, etc.

[0026] In metadata classification, subcategories are divided based on business scenarios and business objects on the basis of the three major categories. For example, under business metadata, metadata can be divided by enterprise, department, commodity, etc., to ensure that the classification is clear and non-overlapping, so as to avoid confusion during classification.

[0027] In step S33, specifically:

[0028] Evaluation indicators include: accuracy, precision, recall, F1 score, and AUC-ROC.

[0029] Accuracy measures the proportion of correct predictions made by the model as a whole, precision measures the proportion of instances predicted to be positive that are actually positive, recall measures the proportion of instances that are correctly predicted that are actually positive, and the F1 score is the harmonic mean of precision and recall. AUC-ROC reflects the performance of the model at different thresholds.

[0030] Dataset division: Before evaluation, the dataset needs to be divided into training set, validation set and test set; the training set is used to train the model, the validation set is used to adjust model parameters and select the best model during the training process, and the test set is used to finally evaluate the performance of the model; ensure that the three datasets are consistent in data distribution and label distribution to avoid the impact of data bias on the evaluation results.

[0031] Model training and validation: In the model training phase, the BERT model is trained using the training set, and the performance of the model is monitored through the validation set. During the training process, the loss function value and accuracy index of each epoch are recorded to observe the training progress and performance changes of the model. At the same time, the hyperparameters of the model, such as learning rate and batch size, are adjusted according to the performance on the validation set to obtain better performance.

[0032] Test set evaluation: After the model training is completed, the test set is used to evaluate the model. The evaluation process includes the following steps: Data preprocessing: Preprocess the metadata in the test set, including text cleaning, word segmentation and tokenization, to ensure that the format of the input data is consistent with that during training. Model prediction: Input the preprocessed metadata into the trained BERT model for prediction; the prediction result is the probability distribution or the most likely label for each category; Calculate evaluation indicators: Based on the prediction results and the true labels of the test set, calculate evaluation indicators such as accuracy, precision, recall, F1 score, and AUC-ROC; the evaluation indicators will be used to measure the performance of the model.

[0033] Result analysis and optimization: After obtaining the results of the evaluation indicators, analyze the results; the analysis content includes but is not limited to:

[0034] Identify performance bottlenecks: Observe which categories have poor classification performance and analyze possible reasons, such as uneven data distribution and insufficient feature extraction.

[0035] Model optimization: Optimize the model based on the analysis results; optimization strategies include adjusting the model structure, adding feature engineering, and using more complex network structures, any one or a combination of these.

[0036] Re-evaluation: After optimization, re-evaluate the model using the test set to verify the optimization effect.

[0037] The specific method of model optimization is any one of the following or a combination of at least two:

[0038] Adjust the convolution kernel size of the model and change the convolution kernel size of the model from (3,3) to (5,5) or (7,7) to observe the impact on the model performance;

[0039] Add batch normalization layers to the model, adding batch normalization layers after convolutional layers and fully connected layers;

[0040] Add a Dropout layer after the fully connected layer of the model and set the Dropout rate (such as 0.55);

[0041] Use a learning rate decay strategy in the model, such as gradually reducing the learning rate as the number of training rounds increases, or using the learning rate adjustment function provided by the optimizer (such as Adam, SGD, etc.);

[0042] The ReLU activation function is used in the convolutional and fully connected layers of the model because it solves the gradient vanishing problem in the positive interval and has fast calculation speed.

[0043] By adopting the above scheme, the present invention provides a method for classifying and labeling database metadata based on artificial intelligence, which involves multiple key steps such as data preparation and preprocessing, model training and optimization, classification and labeling process, and subsequent governance and application by applying artificial intelligence in the process of classifying and labeling database metadata. These steps together constitute a complete process of classifying and labeling database metadata based on artificial intelligence, which provides strong support and guarantee for database management and data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0046] See also Figure 1 The present invention provides a method for classifying and labeling database metadata based on artificial intelligence, which includes the following steps.

[0047] S1: Data preparation and preprocessing.

[0048] Data collection: Extract metadata from the database. Metadata includes table name, field name, data type, description information, etc.; determine the scope and quantity of metadata that needs to be classified and labeled based on actual needs.

[0049] Data cleaning: Clean the extracted metadata to remove invalid, duplicate, incomplete and erroneous data; format the metadata to ensure data consistency and standardization.

[0050] Data annotation: Annotate the cleaned metadata to provide a training set for subsequent machine learning model training.

[0051] S2: Feature extraction.

[0052] Analyze the characteristics of metadata, such as data type, data format, data value range, etc., and extract features that are useful for classification.

[0053] Use natural language processing techniques (such as word segmentation, word frequency statistics, etc.) to extract features from text-type metadata.

[0054] S3: Model training and optimization.

[0055] S31: Select a suitable algorithm model, according to the characteristics of the metadata and classification requirements, select a suitable machine learning algorithm model; preferably, for the metadata name or description, select a basic text similarity algorithm model such as cosine similarity or Jaccard similarity; for the metadata content, select a BERT model suitable for text classification tasks, such as BERT-base-uncased or BERT-base-chinese (if the metadata is in Chinese). The algorithm model is a convolutional neural network model built using a deep learning framework such as Keras or TensorFlow.

[0056] S32: Training model; using the annotated metadata as a training set to train the machine learning model; improving the accuracy and generalization ability of the model by adjusting model parameters, optimizing algorithms, etc.

[0057] Taking the BERT model as an example, the specific method is: Download the model and weights: download the pre-trained BERT model and its weights from model libraries such as Hugging Face; Model fine-tuning: use the annotated metadata to fine-tune the BERT model to adapt to specific classification tasks; During the fine-tuning process, you need to set appropriate hyperparameters, such as learning rate, batch size, etc.

[0058] S33: Use the validation set to evaluate the trained BERT model. The evaluation indicators may include accuracy, recall, F1 score, etc. Based on the evaluation results, optimize and adjust the BERT model to improve the performance of the BERT model.

[0059] S4: Subsequent governance and application; including: data organization and management, data quality monitoring, data analysis and mining. Specifically:

[0060] S41: Data organization and management; Based on the results of classification and labeling, classify and organize the metadata in the database to improve the discoverability and availability of data; use label information to implement data permission control and access management to ensure data security and compliance.

[0061] S42: Data quality monitoring; regularly monitor and analyze the quality of metadata in the database to promptly identify and resolve data quality issues; apply the corresponding annotated metadata (data type, format, value range, etc.) according to the annotated data category, conduct quality inspections and analyze the results; evaluate the changing trends of data quality by comparing labeled data at different time points, and provide decision support for data governance; evaluate data quality improvements based on the proportion of data that do not meet metadata specification requirements at different time points.

[0062] S43: Data analysis and mining: Use the metadata after classification and labeling to conduct more in-depth data analysis and mining to discover the association rules and potential value in the data; understand the association relationship between various business models in the application system that lacks or is missing entity relationships. Break the data silos, cross organizations and systems, integrate different business data, unify data models, and improve data quality.

[0063] The annotation process involves the classification, description, and attribute extraction of metadata;

[0064] Metadata classification is divided into three categories: business metadata (mainly describes the concepts, rules and processes of the business, such as business terms, business rules, data models, etc. It helps business personnel better understand the application and significance of data in the business), technical metadata (describes information such as the physical storage, technical architecture and data processing process of the data, such as database table structure, field type, data storage path, etc. It is crucial for technical personnel to manage and maintain data), and management metadata (information related to data lifecycle management, data quality, data security, etc. It helps managers monitor and manage the entire life cycle of data to ensure the accuracy, integrity and security of data).

[0065] Metadata description refers to the accurate definition of the object represented by the metadata. It cannot be without content and cannot be ambiguous.

[0066] Metadata attributes are the information storage method of metadata: such as data type, data length, precision, format, etc.

[0067] In metadata classification, subcategories are divided based on business scenarios and business objects on the basis of the three major categories. For example, under business metadata, metadata can be divided by enterprise, department, commodity, etc., to ensure that the classification is clear and non-overlapping, so as to avoid confusion during classification.

[0068] In step S33, specifically:

[0069] Evaluation indicators include: accuracy, precision, recall, F1 score, and AUC-ROC.

[0070] Accuracy measures the proportion of correct predictions made by the model as a whole, precision measures the proportion of instances predicted to be positive that are actually positive, recall measures the proportion of instances that are correctly predicted that are actually positive, and the F1 score is the harmonic mean of precision and recall. AUC-ROC reflects the performance of the model at different thresholds.

[0071] Dataset division: Before evaluation, the dataset needs to be divided into training set, validation set and test set; the training set is used to train the model, the validation set is used to adjust model parameters and select the best model during the training process, and the test set is used to finally evaluate the performance of the model; ensure that the three datasets are consistent in data distribution and label distribution to avoid the impact of data bias on the evaluation results.

[0072] Model training and validation: In the model training phase, the BERT model is trained using the training set, and the performance of the model is monitored through the validation set. During the training process, the loss function value and accuracy index of each epoch are recorded to observe the training progress and performance changes of the model. At the same time, the hyperparameters of the model, such as learning rate and batch size, are adjusted according to the performance on the validation set to obtain better performance.

[0073] Test set evaluation: After the model training is completed, the test set is used to evaluate the model. The evaluation process includes the following steps: Data preprocessing: Preprocess the metadata in the test set, including text cleaning, word segmentation and tokenization, to ensure that the format of the input data is consistent with that during training. Model prediction: Input the preprocessed metadata into the trained BERT model for prediction; the prediction result is the probability distribution or the most likely label for each category; Calculate evaluation indicators: Based on the prediction results and the true labels of the test set, calculate evaluation indicators such as accuracy, precision, recall, F1 score, and AUC-ROC; the evaluation indicators will be used to measure the performance of the model.

[0074] Result analysis and optimization: After obtaining the results of the evaluation indicators, analyze the results; the analysis content includes but is not limited to:

[0075] Identify performance bottlenecks: Observe which categories have poor classification performance and analyze possible reasons, such as uneven data distribution and insufficient feature extraction.

[0076] Model optimization: Optimize the model based on the analysis results; optimization strategies include adjusting the model structure, adding feature engineering, and using more complex network structures, any one or a combination of these.

[0077] Re-evaluation: After optimization, re-evaluate the model using the test set to verify the optimization effect.

[0078] The specific method of model optimization is any one of the following or a combination of at least two:

[0079] Adjust the convolution kernel size of the model and change the convolution kernel size of the model from (3,3) to (5,5) or (7,7) to observe the impact on the model performance;

[0080] Add batch normalization layers to the model, adding batch normalization layers after convolutional layers and fully connected layers;

[0081] Add a Dropout layer after the fully connected layer of the model and set the Dropout rate (such as 0.55);

[0082] Use a learning rate decay strategy in the model, such as gradually reducing the learning rate as the number of training rounds increases, or using the learning rate adjustment function provided by the optimizer (such as Adam, SGD, etc.);

[0083] The ReLU activation function is used in the convolutional and fully connected layers of the model because it solves the gradient vanishing problem in the positive interval and has fast calculation speed.

[0084] The present invention provides a method for classifying and labeling database metadata based on artificial intelligence, which is a way to achieve database management and data governance. Database metadata labeling refers to the process of classifying, marking and annotating metadata in the database using artificial intelligence technology. These labels are usually used to indicate key information, attributes or features in metadata, thereby helping users better understand and use these data. The significance of labeling is to improve the discoverability, availability and security of data, while supporting more efficient data analysis and mining.

[0085] The present invention realizes automatic classification and labeling of metadata in the database, reduces manual intervention, and improves processing efficiency. Through machine learning algorithms, the system can learn and identify patterns and features in metadata, and automatically assign correct labels to them. Based on historical data and user behavior, artificial intelligence can recommend appropriate labels to users for reference or selection. At the same time, the system can also predict metadata or labels that users may be interested in, and provide personalized data access and query services. The quality and integrity of database metadata can be monitored in real time, and erroneous or missing labels can be discovered and repaired in a timely manner to improve data quality.

[0086] In summary, the present invention provides a method for classifying and labeling database metadata based on artificial intelligence, which involves multiple key steps such as data preparation and preprocessing, model training and optimization, classification and labeling process, and subsequent governance and application by applying artificial intelligence in the process of classifying and labeling database metadata. These steps together constitute a complete process of classifying and labeling database metadata based on artificial intelligence, which provides strong support and guarantee for database management and data governance.

[0087] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for classifying and labeling database metadata based on artificial intelligence, characterized in that: The following steps are involved: S1: Data preparation and preprocessing; Data collection: Extract metadata from the database, including table name, field name, data type, and description information; Determine the scope and quantity of metadata that need to be classified and labeled based on actual needs; Data cleaning: Clean the extracted metadata to remove invalid, duplicate, incomplete and erroneous data; Format metadata to ensure data consistency and standardization; Data annotation: annotate the cleaned metadata to provide a training set for subsequent machine learning model training; S2: Feature extraction; Analyze the characteristics of metadata and extract features useful for classification; Use natural language processing technology to extract features from text-type metadata; S3: Model training and optimization; S31: Select a suitable algorithm model. According to the characteristics of metadata and classification requirements, select a suitable machine learning algorithm model; S32: training model; Use the annotated metadata as a training set to train the machine learning model; improve the accuracy and generalization ability of the model by adjusting model parameters and optimizing the algorithm; the specific methods are: Download model and weights: Download pre-trained models and their weights; Model fine-tuning: Use the annotated metadata to fine-tune the model to suit a specific classification task; During fine-tuning, you need to set appropriate hyperparameters; S33: Use the validation set to evaluate the trained model; based on the evaluation results, optimize and adjust the model to improve the performance of the model; S4: Subsequent governance and application; including: data organization and management, data quality monitoring, data analysis and mining.

2. According to claim 1, a method for classifying and labeling database metadata based on artificial intelligence is characterized in that: The annotation process involves the classification, description, and attribute extraction of metadata; Metadata classification: classified into three categories: business metadata, technical metadata, and management metadata; Metadata description refers to the accurate definition of the object represented by the metadata. It cannot be without content and cannot be ambiguous. Metadata attributes are the information storage method of metadata.

3. According to claim 2, a method for classifying and labeling database metadata based on artificial intelligence is characterized in that: In metadata classification, subcategories are divided based on business scenarios and business objects on the basis of the three major categories to ensure that the classifications are clear and non-overlapping to avoid confusion during classification.

4. According to the method of claim 1, wherein the method is characterized in that: In step S31, a suitable algorithm model is selected, as follows: For metadata name or description, select the text similarity algorithm model; For metadata content, select a model suitable for the text classification task.

5. According to claim 4, a method for classifying and labeling database metadata based on artificial intelligence is characterized in that: The algorithm model is a convolutional neural network model built using a deep learning framework such as Keras or TensorFlow.

6. According to the method of claim 4, the method is characterized in that: The text similarity algorithm model is a cosine similarity model or a Jaccard similarity model, and the model suitable for text classification tasks is a BERT model.

7. According to claim 1, a method for classifying and labeling database metadata based on artificial intelligence is characterized in that: In step S33, specifically: Evaluation indicators include: accuracy, precision, recall, F1 score and AUC-ROC. Accuracy measures the proportion of correct predictions of the model as a whole, precision measures the proportion of instances predicted as positive samples that are actually positive samples, recall measures the proportion of instances that are correctly predicted that are actually positive samples, F1 score is the harmonic mean of precision and recall, and AUC-ROC reflects the performance of the model at different thresholds. Dataset division: Before evaluation, the dataset needs to be divided into training set, validation set and test set. The training set is used to train the model, the validation set is used to adjust the model parameters and select the best model during the training process, and the test set is used to finally evaluate the performance of the model. Ensure that the three datasets are consistent in data distribution and label distribution to avoid the impact of data bias on the evaluation results. Model training and verification: In the model training phase, the model is trained using the training set, and the performance of the model is monitored through the verification set. During the training process, the loss function value and accuracy index of each epoch are recorded to observe the training progress and performance changes of the model. At the same time, the hyperparameters of the model are adjusted according to the performance on the verification set to obtain better performance. Test set evaluation: After the model training is completed, the model is evaluated using the test set. The evaluation process includes the following steps: Data preprocessing: Preprocess the metadata in the test set to ensure that the format of the input data is consistent with that during training; Model prediction: The preprocessed metadata is input into the trained model for prediction; the prediction result is the probability distribution or the most likely label for each category; Calculate evaluation indicators: Calculate evaluation indicators based on the prediction results and the true labels of the test set; the evaluation indicators will be used to measure the performance of the model; Result analysis and optimization: After obtaining the results of the evaluation indicators, analyze the results; the analysis content includes but is not limited to: Performance bottleneck identification: observe which categories have poor classification performance and analyze possible reasons; Model optimization: Optimize the model based on the analysis results; optimization strategies include adjusting the model structure, adding feature engineering, and using more complex network structures, any one or a combination of these; Re-evaluation: After optimization, re-evaluate the model using the test set to verify the optimization effect.

8. The method for classifying and labeling database metadata based on artificial intelligence according to claim 7, characterized in that: The specific method of model optimization is any one of the following or a combination of at least two: Adjust the convolution kernel size of the model and change the convolution kernel size of the model from (3,3) to (5,5) or (7,7) to observe the impact on the model performance; Add batch normalization layers to the model, adding batch normalization layers after convolutional layers and fully connected layers; Add a Dropout layer after the fully connected layer of the model and set the Dropout rate; Use a learning rate decay strategy in your model; The ReLU activation function is used in the convolutional and fully connected layers of the model.

9. The method for classifying and labeling database metadata based on artificial intelligence according to claim 1, characterized in that: The step S4 is specifically: S41: Data organization and management: Based on the classification and labeling results, the metadata in the database is classified and organized to improve the discoverability and availability of data; using label information, data authority control and access management are implemented to ensure data security and compliance; S42: Data quality monitoring; Regularly monitor and analyze the quality of metadata in the database to promptly identify and resolve data quality issues; According to the annotated data category, apply the corresponding annotated metadata to perform quality inspection and analyze the results; By comparing label data at different time points, we can evaluate the changing trend of data quality and provide decision support for data governance. We can also evaluate the improvement of data quality based on the proportion of data that do not meet metadata specification requirements at different time points. S43: Data analysis and mining: Use the classified and labeled metadata to conduct more in-depth data analysis and mining to discover association rules and potential value in the data.

Images