Public data classification and grading identification method based on AI large model

Abstract

The invention discloses an AI large model-based public data classification and grading identification method, which comprises the following steps of: S1) pre-analyzing a supported file format by adopting a file analysis engine to form a classification and grading standard file; s2) the large model base adopts an enhanced retrieval generation process, and a data classification and grading database is established; s3) inputting an original data table and unstructured data, and completing initial data summarization of the large model; s4) adopting Few-Shot to learn successful classification cases memorized for a long time, and performing large model question and answer; and S5) based on all possible large model question and answer classification results, the large model engine performs automatic question and answer again, autonomous reflection is performed based on a first round of intelligent classification result, and a new data classification result and classification confidence are determined again. According to the method, the recognition accuracy can be greatly improved, and the problems of low accuracy, low efficiency, many human errors, poor adaptability and the like in a traditional data classification and grading method are solved.

Description

Technical Field

[0001] The present invention relates to a method for classifying and grading public data recognition, and particularly to a method for classifying and grading public data based on an AI large model. Background Art

[0002] Currently, public data of digital governments everywhere are facing challenges in difficult discovery and recognition of classification and grading. Public data are widespread and diverse in form, requiring support for discovering data with different storage structures on various data source types, and the data scale is often large, requiring sorting out massive amounts of data, presenting great technical difficulties. For the classification and grading work of public data, according to standards such as the "Rules for Data Classification and Grading", the data needs to be divided into categories such as personal data, user data, and business data, and further sub-graded to achieve differential management and protection.

[0003] Existing classification and grading technologies generally rely on rule engines such as certain keyword inclusion rules, regular expressions, fuzzy matching, dictionary matching rules, and key-value pair inclusion rules to achieve. For rules with obvious features and easy to abstract unified recognition features, this method is feasible. However, in the case of incomplete data governance, due to the easy deformation and combination of data, the problem of accurate recognition of single-field data and the problem of data levels brought by multi-field combination become complex; currently, the compliance standards for classification and grading are not unified, the features are inaccurate, and automatic learning is not possible. At the same time, public data sources are extensive, including data from all walks of life and fields, covering personal identity information, education information, social security data, travel information, etc. The data volume is large, the dimensions are many, the data forms and standards are not unified, and they are numerous and miscellaneous. Therefore, how to improve the accuracy of public data classification and grading has become a key and difficult task in the current data security construction of digital governments. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for classifying and grading public data recognition based on an AI large model, which can greatly improve the recognition accuracy, thus significantly solving and improving problems such as low accuracy, low efficiency, many human errors, and poor adaptability existing in traditional data classification and grading methods.

[0005] The technical solution adopted by the present invention to solve the above technical problems is to provide a recognition method for the classification and grading of public data based on an AI large model, including the following steps: S1) Use a file parsing engine to pre-parse the supported file formats to form a classification and grading specification file, output the main information in plain text, and store it in a vector database; S2) The large model base uses an enhanced retrieval generation process to obtain relevant category definition information from the vector database and establish a database for data classification and grading; S3) Input the original data table and unstructured data, and judge the data types in the data table through large model questions and answers to complete the initial data summary of the large model; S4) Adopt successful classification cases of Few-Shot learning long-term memory, enhance the matching degree of classification and grading retrieval results through the Embedding deep learning model, conduct large model questions and answers, and find appropriate classification question and answer results and classification confidence levels; S5) Based on all possible large model question and answer classification results, the large model engine conducts automatic questions and answers again, conducts self-reflection based on the first-round intelligent classification results, and re-determines the new data classification and grading results and classification confidence levels.

[0006] Further, the step S1 includes: If it is a picture, use the OCR engine to convert the picture into text, and then use regular expressions and word segmentation methods according to the text to extract the occurrence frequency and attributes of fine-grained sensitive keywords.

[0007] Further, for the recognition of variable-length text in the step S1: Extract image features through a convolutional layer, and then process sequence features through a recurrent neural network; and introduce an attention mechanism to process long-sequence data to improve the recognition accuracy; then define sensitive information according to the data security policy, and use regular expressions or pre-trained NER models to identify sensitive information in the text.

[0008] Further, the step S1 also includes realizing sensitive data matching by combining context information, identifying the relationships between entities through relation extraction technology, and further confirming whether the extracted entities are sensitive data; and using a model based on the Softmax neural network to conduct intelligent classification and grading of customer sensitive information; The intelligent classification and grading method is realized through two full-scale data asset scanning operations with different focuses: The first scan: Conduct preliminary classification and grading of the full-scale data assets to identify potential sensitive information; In this stage, use an unsupervised learning model to classify and grade a large number of unlabeled samples through clustering, and accumulate initial labeled samples; The second scan: On the basis of the first scan, use a Softmax classifier to classify the feature vectors to achieve more accurate classification and grading.

[0009] Further, the file formats supported in step S1 include XLS, XLSX, PDF, DOC, DOCX, PPT, PPTX, XML, JSON, and DAT format files; the original data table in step S3 comes from mysql, hive, or redis, and the unstructured data comes from doc, txt, or pdf.

[0010] Further, the large model is the llama3 large model, and the data training performed by the llama3 large model includes: data cleaning, data verification, data annotation, data source selection, data version control, and regular review.

[0011] Further, the llama3 large model uses accuracy, recall, mean squared error, and coefficient of determination as evaluation metrics to measure the performance of the model, and continuously optimizes the model according to the feedback of the evaluation metrics in the following ways; Adjust the model structure: increase or decrease the number of layers, change the number of hidden units, introduce the attention mechanism, switch from traditional convolutional neural networks to the Transformer architecture, and introduce residual connections or skip connection structures in the model; Optimize the training strategy: use a fixed learning rate for simple tasks, gradually decrease the learning rate using exponential decay and cosine decay as training progresses, and dynamically adjust the learning rate according to the magnitude of the gradient; Adjust the algorithm parameters: use grid search or random search, dynamically adjust the search direction through the Bayesian method to find the optimal parameter combination, and combine multiple models through ensemble learning methods.

[0012] Further, the optimization training strategy also includes: making the model parameters sparse by penalizing the absolute value of the model parameters to reduce the complexity of the model; restricting the magnitude of the parameters by penalizing the square value of the model parameters to prevent overfitting; enhancing the generalization ability of the model by randomly dropping some neurons; in image processing, increasing the diversity of data through rotation, cropping, and flipping operations; in natural language processing, increasing the diversity of text data through synonym replacement and random insertion operations, thereby improving the generalization ability of the model.

[0013] Further, the llama3 large model uses the training set and the test set to evaluate the model performance, and uses the validation set for cross-validation. The proportion of the training set is 80%, the proportion of the validation set is 10%, and the proportion of the test set is 10%; during the training process, when the performance of the model on the validation set no longer improves, prevent overfitting by reducing the learning rate or adjusting the regularization strength.

[0014] Further, the llama3 large model reduces the storage and computational overhead of the model while maintaining the model performance through pruning, quantization, and knowledge distillation.

[0015] The present invention has the following beneficial effects compared with the prior art: The recognition method for the classification and grading of public data based on the AI large model provided by the present invention introduces the Llama3 + RAG technology into the sensitive data discovery engine to annotate the currently discovered sensitive data, and transfers the annotated data flow to the large model inference service for further data analysis to obtain the data classification and grading results; it can greatly improve the recognition accuracy, thus significantly solving and improving the problems of low accuracy, low efficiency, many human errors, and poor adaptability existing in the traditional data classification and grading methods. This not only helps to improve the management and utilization level of public data, but also injects new vitality into the development of data security governance of digital government. Brief Description of the Drawings

[0016] Figure 1 It is a schematic diagram of the recognition of the classification and grading of public data based on the AI large model of the present invention; Figure 2 It is a schematic diagram of the accuracy management of the AI large model of the present invention; Figure 3 It is the analysis result of a table by the large model of the present invention. Detailed Embodiments

[0017] The present invention will be further described below in conjunction with the drawings and embodiments.

[0018] Traditional data classification and grading methods may have problems of recognition errors or inaccurate classification, especially when dealing with complex and diverse public data. The AI large model can significantly improve the accuracy of data classification and grading through deep learning algorithms and advanced model training techniques. In the existing traditional classification and grading techniques, especially for unstructured data, the efficiency of data classification and grading is low, and sometimes manual intervention is required for judgment, which may lead to classification errors due to fatigue, inattention, etc. The AI large model is not affected by these factors and can continuously, stably and efficiently perform data classification and grading, thus reducing the occurrence of human errors and being able to adjust and optimize the model according to new data features or requirements.

[0019] Please refer to Figure 1 , according to the classification and grading specifications of public data and the application capabilities of the AI large model technology, the present invention realizes the intelligent recognition of structured data and unstructured / semi-structured data, and completes the sensitive recognition and classification and grading of structured data and unstructured / semi-structured data in the database resources accessed on the platform side. It mainly includes three core functions: a summary module, a classification module, and a reflection module: Summary module: Input the original data tables (such as mysql, hive, redis, etc.) and unstructured data (such as doc, txt, pdf, etc.), and determine the data types in the data tables through large model Q&A to complete the initial data summary of the large model.

[0020] Classification module: Adopt successful classification cases of Few-Shot learning long-term memory, enhance the matching degree of classification and grading retrieval results through the Embedding deep learning model (RAG process), conduct large model Q&A, and give all possible intelligent classification Q&A results and classification confidence levels.

[0021] Reflection module: Based on all possible large model Q&A classification results, the large model engine conducts automatic Q&A again, conducts independent reflection based on the first-round intelligent classification results, re-determines the new data classification and grading results and classification confidence levels, and achieves the effect of improving the accuracy of data recognition.

[0022] Among them, the file parsing engine parses the format of the pre-parsed supported files and outputs the main information in plain text. If it is a picture, the OCR engine is used to convert the picture into text, and then according to the text, regular expressions and word segmentation methods are used to extract fine-grained sensitive information and attributes, the occurrence frequency and attributes of fine-grained sensitive keywords, such as: the number of occurrences of ID card numbers, the address where the ID card number is located, date of birth, gender and other attributes.

[0023] The large model base adopts the enhanced retrieval generation (RAG) process to obtain relevant category definition information from the vector database and establish a database for data classification and grading. The summary module retrieves multiple theme information of the file content through Q&A with the llama3 large model. The classification module finds the appropriate classification and data types through Q&A with the llama3 large model (note: here the large model will automatically trigger RAG linkage and pre-match from the classification and grading rules). The reflection module determines the appropriate best classification and data types from the data types obtained from the classification, similar to scoring a more suitable classification.

[0024] The integration of traditional rule engines and AI large model engines can combine the advantages of both to achieve more efficient and accurate data classification and grading. Specifically, the rule engine can be used to handle simple and clear business rules, while the large model engine can be used to handle complex and fuzzy data patterns. Classification and grading is a dynamic process that needs to be continuously adjusted and optimized as the business develops and technology progresses. The effectiveness and existing problems of classification and grading should be reviewed regularly, and the classification criteria and security measures should be adjusted in a timely manner; attention should also be paid to the development and application of new technologies, and new tools and methods should be continuously introduced to improve the efficiency and accuracy of classification and grading.

[0025] Please continue to refer to Figure 2, the accuracy of the AI model depends on efficient data training, precise metric evaluation, dataset splitting, model compression, distributed and parallel computing, and continuous model optimization. These core functions together constitute a solid foundation for the accuracy of the AI model, ensuring that the model can provide reliable and efficient prediction results in various application scenarios.

[0026] 1) Efficient data training: including data cleaning (removing noise, handling missing values), data validation (manual inspection, automated detection), data annotation, data source selection (reliable sources, diverse sources), data version control (recording changes, traceability), and regular reviews (periodic checks, user feedback).

[0027] 2) Precise metric evaluation: Measuring the performance of the model by defining appropriate evaluation metrics, such as accuracy, recall, mean squared error (MSE), coefficient of determination (R²), etc.

[0028] 3) Dataset splitting: Using training sets and test sets to evaluate the model performance, such as cross-validation methods, which can reduce the evaluation bias caused by a single dataset split.

[0029] 4) Model compression: Reducing the storage and computational overhead of the model while maintaining its performance through techniques such as pruning, quantization, and knowledge distillation.

[0030] 5) Distributed and parallel computing: Utilizing distributed computing and parallel computing technologies to accelerate the model training process, especially when dealing with large-scale datasets.

[0031] 6) Continuous model optimization: Continuously optimizing the model based on the feedback of evaluation metrics, including adjusting the model structure, training strategy, and algorithm parameters.

[0032] i) Ways to adjust the model structure: Increasing or decreasing the number of layers: Increasing the number of layers: Increasing the depth of the model can enhance its expressive power, enabling it to learn more complex features. However, an overly deep model may lead to increased training time and the risk of overfitting.

[0033] Decreasing the number of layers: Decreasing the depth of the model can reduce the computational burden, but it may also degrade the model's performance.

[0034] Changing the number of hidden units: The number of hidden units determines the capacity of each layer. Too many hidden units may lead to overfitting, while too few may not be able to fully learn the features in the data.

[0035] Introducing the attention mechanism: The attention mechanism can help the model better focus on the important parts of the input data, thereby improving the performance of the model. For example, in natural language processing tasks, the Transformer architecture effectively models long-range dependencies through the self-attention mechanism.

[0036] Adjust the network architecture: For example, switch from a traditional convolutional neural network (CNN) to the Transformer architecture, or introduce structures such as residual connections and skip connections in the model to improve the stability of gradient propagation and model training.

[0037] ii) Adjustment methods for optimizing training strategies: Learning rate adjustment: Fixed learning rate: Keep the learning rate unchanged throughout the training process, which is suitable for simple tasks.

[0038] Learning rate decay: Gradually reduce the learning rate as training progresses. Common methods include exponential decay, cosine decay, etc.

[0039] Adaptive learning rate methods: Such as the Adam optimizer, which can dynamically adjust the learning rate according to the magnitude of the gradient, improving the efficiency and stability of training.

[0040] Batch size adjustment: Batch size: Determines the amount of data used when updating model parameters each time. A larger batch size can improve the stability of training, but will increase memory requirements and computational overhead; a smaller batch size can reduce memory usage, but may increase noise during training and affect the convergence speed of the model.

[0041] Regularization methods: L1 regularization: By penalizing the absolute value of model parameters, it sparsifies the model parameters and reduces the complexity of the model.

[0042] L2 regularization: By penalizing the squared value of model parameters, it limits the magnitude of the parameters and prevents overfitting.

[0043] Dropout: By randomly discarding some neurons, it enhances the generalization ability of the model.

[0044] Data augmentation: In image processing, data diversity can be increased through operations such as rotation, cropping, and flipping.

[0045] In natural language processing, text data diversity can be increased through operations such as synonym replacement and random insertion, thereby improving the generalization ability of the model.

[0046] iii) Adjustment methods for adjusting algorithm parameters: Hyperparameter Search: Grid Search: By exhaustively enumerating all possible parameter combinations, the optimal parameter configuration is found.

[0047] Random Search: Randomly select parameter combinations in the parameter space for search, which is usually more efficient than grid search.

[0048] Bayesian Optimization: Dynamically adjust the search direction through Bayesian methods to find the optimal parameter combination.

[0049] Dynamic Adjustment Strategy: During the training process, adjust the parameters dynamically according to the performance of the model. For example, when the performance of the model on the validation set no longer improves, the learning rate can be reduced, or the regularization strength can be adjusted to prevent overfitting.

[0050] Model Fusion: By using ensemble learning methods such as Bagging, Boosting, or Stacking, multiple models are combined to improve the overall performance. This method can utilize the advantages of different models and reduce the bias and variance of a single model.

[0051] It has been connected to the business system for testing, covering important-level systems. The real-time data training data is 1TB, the total training data is 30B, the proportion of the training set is 80%, the validation set is 10%, and the test set is 10%. The final achievable effects are: text classification accuracy: 96.5%, image recognition accuracy: 92.3%.

[0052] The implementation results of the recognition of public data classification and grading based on the AI large model of the present invention are as follows: 1. Standardization of data classification and grading rules Implementation results: By introducing large models and intelligent technologies, the unified naming and annotation specifications of data assets have been realized, effectively solving the problems of data classification and grading caused by chaotic naming and non-standard annotations. The standardized data assets are convenient for management and maintenance, and the data utilization efficiency has been improved. In the practice of classification and grading standardization rules, it will be found that: 1) There are ambiguous semantics in the specification description part; 2) There is some overlapping space between categories; 3) The coverage of the specification is limited, and a large number of fields cannot find classification and data types; Make full use of the understanding, discovery, and naming capabilities of large models to promote the development of specifications from bottom to top; make the specifications more reasonable and have a wider coverage.

[0053] Figure 3It is the analysis result of a large model for a table. The renamed Chinese names and category names are the analysis results of the large model. Among them, "Time and Date" is the category name defined by the large model.

[0054] 2. The recognition rate has been significantly improved Implementation effect: By using a large number of data training sets for unsupervised multi-objective learning, the recognition rate of data classification and grading has been significantly improved. The initial accuracy of the rule engine is on average 20% - 30%, and the initial accuracy of the large model is estimated to be on average 90%. In contrast, the large model recognition engine, through means such as deep learning technology, feature extraction, context understanding and reasoning, has an initial accuracy more than 3 times higher than that of the rule engine.

[0055] 3. The coverage is more comprehensive In addition to regular structured data, new parsing of rich text data such as HTML, PDF, Word documents, etc. is added. These formats not only contain text content but may also contain complex elements such as pictures, tables, styles, etc. The key to parsing rich text data is to extract the pure text content while preserving its semantic structure. HTML parsing: Tools such as BeautifulSoup (Python library) can be used to extract the text content in HTML and remove HTML tags. PDF and Word document parsing: Tools such as Apache Tika or PDFMiner can be used to extract the text in PDF and Word documents. By combining named entity extraction and character recognition algorithms, named entities with specific meanings such as person names, place names, and organization names can be recognized from the text; for the picture content in rich text data, the character recognition algorithm (OCR) can convert the text in the picture into editable text. Common OCR algorithms include CRNN (Convolutional Neural Network + Recurrent Neural Network) which is suitable for variable-length character recognition, extracting image features through convolutional layers and then processing sequence features through recurrent neural networks; CNN + Seq2Seq + Attention (introducing the attention mechanism), which can better process long sequence data and improve recognition accuracy. Then, according to the data security policy, define which types of entities (such as ID card numbers, bank card numbers, person names, etc.) belong to sensitive information, and use regular expressions or pre-trained NER models to identify sensitive information in the text, or combine context information to further confirm whether the extracted entity is sensitive data. For example, through relation extraction technology to identify the relationships between entities, so as to more accurately judge the sensitivity of information, achieve sensitive data matching, and achieve the effect of automatically discovering high-sensitive data types and expanding the scope and types of sensitive data recognition. By supporting the recognition ability of file formats such as XLS, XLSX, PDF, DOC, DOCX, PPT, PPTX, XML, JSON, DAT (including pure text data and picture data), the file coverage rate of sensitive data discovery and classification and grading is comprehensively improved.

[0056] 4. Enhanced timeliness By continuously optimizing the model algorithm and improving computing power, after the large model is tuned, the reasoning and explanation auxiliary effect can more significantly improve the timeliness of classification and grading. This invention effectively utilizes the classification and grading strategy research and key technologies based on the large model, and its advancement is specifically reflected in the following aspects: 1. RAG’s retrieval capability and Few-Shot Learning’s efficient learning capability 1) Deep learning and memory mechanism: The classification subgraph uses Few-Shot learning and deep learning models, such as Embedding, which can efficiently learn and memorize based on a small amount of training data, improving the accuracy of classification and retrieval. The RAG process enhances the matching degree of the model to the data, making the classification results more accurate.

[0057] 2) Efficient file parsing and information extraction: The file parsing engine can quickly parse files in various formats, convert image content into text through OCR technology, and then use regular expressions and word segmentation methods to extract key information. This fine-grained sensitive information extraction capability makes data processing more in-depth and comprehensive.

[0058] 3) Enhanced retrieval generation of large model base: Through the RAG process, the large model can retrieve key category definition information from the vector database and build a database of data classification and grading. This not only improves the efficiency of data classification, but also enables the model to better understand and apply classification rules.

[0059] 4) Intelligent classification and grading: Use a model based on the Softmax neural network to achieve intelligent classification and grading of customer sensitive information in the massive unstructured text data in public data. The intelligent classification and grading method is implemented through two full-data asset scanning operations with different focuses: The first scan: Preliminary classification and grading of the full-data assets to identify potential sensitive information. In this stage, an unsupervised learning model can be used to classify and grade a large number of unlabeled samples through clustering and other methods to accumulate initial labeled samples. The second scan: Based on the first scan, the supervised learning model is used to further improve the accuracy of classification and grading. For example, the Softmax classifier is used to classify feature vectors to achieve more accurate classification and grading. This two-scan operation method can not only improve the efficiency of classification and grading, but also ensure the accuracy of the results and meet the needs of data security management. The basic information of unstructured data and the sample data are matched with asset identification and classification and grading rules to ensure the automatic classification and grading of sensitive data assets in public data and the verification of desensitization effects.

[0060] 2. Large model prediction introduces a reflection process 1) Diversity of fine-tuning strategies: Through strategies such as data cleaning, targeted fine-tuning, RAG integration, few-shot learning, model continuous learning, and manual review, large models can efficiently adapt to specific tasks, reduce false positives, and improve accuracy. These strategies combine the latest developments in modern machine learning techniques, demonstrating the high flexibility and adaptability of the models.

[0061] 2) Multimodal processing ability: RAG supports multimodal content processing, capable of handling complex information containing various types of data, enhancing the model's comprehensive ability in dealing with complex real-world problems.

[0062] 3) Powerful pre-training foundation: GPT series models are pre-trained on a large amount of text data, which endows them with powerful language understanding capabilities. This pre-training ability provides a solid foundation for fine-tuning, enabling the model to quickly adapt to specific classification and grading tasks.

[0063] 4) Adaptability and flexibility: Through fine-tuning, it can adapt to the vast amount of unstructured data in public data. During the fine-tuning process, the model can learn task-specific patterns and features, thereby improving the effect of classification and grading.

[0064] Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the claims.

Claims

1. An identification method for the classification and grading of public data based on a large AI model, characterized in that, It includes the following steps: S1) Use a file parsing engine to pre-parse the supported file formats to form a classification and grading specification file, output the main information in plain text, and store it in a vector database; S2) The large model base uses an enhanced retrieval and generation process to obtain relevant category definition information from the vector database and establish a database for data classification and grading; S3) Input the original data table and unstructured data, and use the large model to answer questions to determine the data types in the data table to complete the initial data summary of the large model; S4) Adopt successful classification cases of Few-Shot learning for long-term memory, enhance the matching degree of classification and grading retrieval results through the Embedding deep learning model, conduct large model question answering, and find appropriate classification question answering results and classification confidence levels; S5) Based on all possible large model question answering classification results, the large model engine conducts automatic question answering again, conducts self-reflection based on the first-round intelligent classification results, and re-determines the new data classification and grading results and classification confidence levels.

2. The recognition method for classification and grading of public data based on the AI large model according to claim 1, wherein, The step S1 includes: if it is a picture, use the OCR engine to convert the picture into text, and then use regular expressions and word segmentation methods to extract the occurrence frequency and attributes of fine-grained sensitive keywords according to the text.

3. The recognition method for classifying and grading public data based on the AI large model according to claim 2, wherein, The step S1 identifies the following for variable-length text: extract image features through the convolutional layer, and then process the sequence features through the recurrent neural network; And introduce the attention mechanism to process long sequence data to improve the recognition accuracy; then define sensitive information according to the data security policy, and use regular expressions or pre-trained NER models to identify sensitive information in the text.

4. The recognition method for classification and grading of public data based on the AI large model according to claim 3, wherein, The step S1 further includes implementing sensitive data matching by combining context information, identifying the relationships between entities through relation extraction technology, and further confirming whether the extracted entities are sensitive data; and using a model based on the Softmax neural network to perform intelligent classification and grading of customer sensitive information; The intelligent classification and grading method is realized through two full-scale data asset scanning operations with different focuses: the first scan: conduct a preliminary classification and grading of the full-scale data assets to identify potential sensitive information; in this stage, use an unsupervised learning model to classify and grade a large number of unlabeled samples through clustering, and accumulate initial labeled samples; the second scan: based on the first scan, use a Softmax classifier to classify the feature vectors to achieve more accurate classification and grading.

5. The recognition method for classification and grading of public data based on the AI large model according to claim 1, characterized in that, The supported file formats in the step S1 include files in the XLS, XLSX, PDF, DOC, DOCX, PPT, PPTX, XML, JSON, DAT formats; the original data table in the step S3 comes from mysql, hive or redis, and the unstructured data comes from doc, txt or pdf.

6. The recognition method for classification and grading of public data based on the AI large model according to claim 1, wherein, The large model is the llama3 large model, and the data training conducted by the llama3 large model includes: data cleaning, data verification, data annotation, data source selection, data version control, and regular review.

7. The recognition method for classification and grading of public data based on the AI large model according to claim 6, wherein, The Llama 3 large model uses accuracy, recall, mean squared error, and coefficient of determination as evaluation metrics to measure the performance of the model, and continuously optimizes the model according to the feedback of the evaluation metrics as follows; Adjust the model structure: increase or decrease the number of layers, change the number of hidden units, introduce the attention mechanism, switch from a traditional convolutional neural network to the Transformer architecture, and introduce residual connections or skip connection structures in the model; Optimize the training strategy: use a fixed learning rate for simple tasks, gradually decrease the learning rate using exponential decay or cosine decay as training progresses, and dynamically adjust the learning rate according to the magnitude of the gradient; Adjust the algorithm parameters: use grid search or random search, dynamically adjust the search direction through Bayesian methods to find the optimal parameter combination, and combine multiple models through ensemble learning methods.

8. The recognition method for classification and grading of public data based on the AI large model according to claim 7, characterized in that, The optimization training strategy also includes: By penalizing the absolute value of the model parameters, sparsify the model parameters and reduce the complexity of the model; By penalizing the squared value of the model parameters, limit the magnitude of the parameters to prevent overfitting; By randomly dropping out some neurons, enhance the generalization ability of the model; In image processing, increase the diversity of data through rotation, cropping, and flipping operations; In natural language processing, increase the diversity of text data through synonym replacement and random insertion operations, thereby improving the generalization ability of the model.

9. The recognition method for classification and grading of public data based on the AI large model according to claim 6, wherein, The Llama 3 large model uses the training set and the test set to evaluate the model performance, and uses the validation set for cross-validation. The proportion of the training set is 80%, the proportion of the validation set is 10%, and the proportion of the test set is 10%; during the training process, when the performance of the model on the validation set no longer improves, prevent overfitting by reducing the learning rate or adjusting the regularization strength.

10. The identification method for public data classification and grading based on the AI large model according to claim 6, characterized in that, The Llama 3 large model reduces the storage and computational overhead of the model while maintaining the model performance through pruning, quantization, and knowledge distillation.

Images