Tobacco industry document automatic classification and storage method using deep learning

Through the fine-tuning and pre-processing technology of the RoBERTa model, the problem of inefficient document management and retrieval in the tobacco industry is solved, efficient document classification and similar document retrieval are achieved, and the accuracy and user experience of the knowledge base are improved.

CN118916484AInactive Publication Date: 2024-11-08SHANDONG JINING TOBACCO CO LTD

Patent Information

Application Number
CN202410841894.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The management and sharing mechanism of the tobacco industry rely on manual transmission, resulting in inefficiency. The lack of effective research on deep learning technology in the application of knowledge bases in specific industries, making it difficult to achieve efficient document classification and retrieval.

Method used

The RoBERTa pre-trained model is used to automatically classify and store documents in tobacco industry. Through model selection, pre-training, data preparation and fine-tuning, combined with Masked Language Model and dynamic mask generation, word segmentation processing, entity recognition and labeling of documents are carried out to achieve word segmentation processing, entity recognition and labeling of documents to achieve Intelligent search and knowledge base construction.

Benefits of technology

It significantly improves the accuracy and efficiency of document classification and similar document retrieval, realizes efficient knowledge base construction and management innovations, and improves user experience and knowledge sharing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916484A_ABST
    Figure CN118916484A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of deep learning, and discloses a method for automatically classifying and storing tobacco industry documents by using deep learning, which comprises the following steps of: selecting a model; the model pre-training process comprises an MLM stage and a dynamic mask generation stage; data preparation and preprocessing: data collection, document format conversion, text cleaning, word segmentation processing, stop word removal and data enhancement; a model fine tuning strategy: defining a task-specific model architecture, selecting a proper loss function and optimizer, and setting and adjusting a strategy for parameters; a model fine tuning process: data batch processing and an early stop method; performing entity identification and tagging; performance evaluation: data set construction, performance evaluation index setting, comparison model setting and experiment environment setting; the experimental results comprise a document classification task, a similar document retrieval task and training and reasoning time. According to the method, the RoBERTa pre-training model is finely tuned, so that the RoBERTa pre-training model has strong context understanding capability, flexible fine tuning capability and efficient data processing capability, and the user experience is also improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for automatically classifying and storing tobacco industry documents using deep learning. Background Art

[0002] With the remarkable achievements of tobacco in management innovation, thousands of excellent innovative achievements have been accumulated through years of hard work. These achievements are not only the crystallization of organizational wisdom, but also a valuable resource for promoting the sustainable development of enterprises. However, the current management and sharing mechanism of these innovative achievements is still insufficient, mainly relying on manual point-to-point transmission, resulting in low efficiency in the use of achievements and difficulty in realizing their maximum value.

[0003] In the current information age, enterprises and organizations are faced with the challenge of how to effectively manage and utilize growing data resources. Especially in knowledge-intensive industries such as the tobacco industry, there are a large number of underutilized document resources, which is crucial to maintaining the innovation ability and competitive advantage of enterprises.

[0004] In recent years, deep learning technologies, especially pre-trained language models such as BERT and RoBERTa, have made remarkable progress in the field of natural language processing. These technologies have shown excellent performance in tasks such as text classification and entity recognition, providing a new way to optimize knowledge base systems. Although these models perform well in general language processing tasks, their application research in knowledge bases of specific industries is relatively insufficient, especially in the document processing of management innovation results in the tobacco industry. At present, there is a lack of in-depth research on how to apply these advanced deep learning technologies to knowledge management and retrieval systems in specific industries. There are still problems that have not been fully addressed, such as how to efficiently build and maintain knowledge bases, and how to improve the accuracy and efficiency of knowledge retrieval. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and propose a method for automatic classification and storage of tobacco industry documents using deep learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: The automatic classification and storage method of tobacco industry documents using deep learning includes the following specific steps: S1: Model selection: Considering the large amount of Chinese documents in the tobacco industry and the need to accurately understand complex management knowledge, RoBERTa was selected as the basis for the pre-training model. RoBERTa's efficient information processing mechanism and excellent understanding of Chinese corpus make it an ideal choice for building a knowledge base and realizing intelligent search functions; S2: Model pre-training process: S2a: Masked Language Model (MLM) stage: The model randomly masks a part of the vocabulary in the input text and then predicts these masked words, learning the deep semantic relationships of the language in this way; S2b: Dynamic mask generation stage: Different from BERT, RoBERTa dynamically changes the masked words during each data pass, increasing the difficulty and diversity of model training; S3: Data preparation and preprocessing: S3a: Data collection: Collect industry documents from the internal database of the tobacco company; S3b: Document format conversion: Use automated scripts to convert all non-text format documents into plain text format, removing formatting elements for subsequent processing; S3c: Text cleaning: Clean the converted text content, remove irrelevant information, and at the same time remove or replace special characters and unstructured information in the text; S3d: Word segmentation processing: Considering the characteristics of Chinese documents, use a deep learning-based Chinese word segmentation tool for word segmentation processing to ensure that the text is correctly segmented into words or phrases, and standardize all text to lowercase (for English parts) to unify the data format; S3e: Stop word removal: According to the predefined stop word list, remove common stop words in the text, such as "的", "是", "在", etc. Although these words are used frequently, they contribute limitedly to understanding the meaning of the document; S3f: Data augmentation: To avoid data bias during model training, adopt a data augmentation strategy. For example, generate new text instances through methods such as synonym replacement and sentence recombination, which can help the model learn more generalized language features and improve performance on different types of documents; S4: Model fine-tuning strategy: S4a: Define a task-specific model architecture: For knowledge base construction, it usually involves tasks such as document classification or similar document retrieval. Therefore, add one or more fully connected layers on top of the RoBERTa language model as the output layer to adapt to these tasks; S4b: Select appropriate loss functions and optimizers: According to the different tasks, select appropriate loss functions to guide model training; for classification tasks, use the cross-entropy loss function; for similarity matching tasks, use cosine similarity as the loss function; at the same time, select an effective optimizer (such as AdamW) to adjust the model parameters to achieve better training results; S4c: Parameter setting and adjustment strategy: Set and adjust the learning rate, batch size, and number of training epochs; S5: Model fine-tuning process: The preprocessed document data is input into the RoBERTa model, and the loss is calculated based on the output of the task-specific layer, and then the model parameters are adjusted through the back-propagation algorithm; S5a: Data batching: Since document datasets can be quite large, it is usually necessary to input data into the model in batches, which can not only improve memory utilization but also help the generalization ability of the model; S5b: Early Stopping: To avoid overfitting, you can terminate the training early when the performance of the validation set no longer improves. This method can help us find a balance between improving model performance and preventing overfitting. S6: Entity recognition and labeling: After the model is fine-tuned, the fine-tuned RoBERTa model is used to perform named entity recognition tasks. By defining a set of entity types for management innovation results documents, the model can automatically identify key information in the document and mark it as the corresponding entity, providing support for automatic labeling of the document; S7: Performance evaluation: S7a: Dataset construction: From the management innovation achievement documents collected by tobacco companies, 80% are randomly selected as training sets, 10% as validation sets, and the remaining 10% as test sets to ensure that the dataset covers a wide range of management innovation topics and content types; S7b: Performance evaluation indicators: In order to comprehensively evaluate the performance of the model, several common evaluation indicators were selected, including precision, recall, F1 score, and accuracy of the model on the test set. These indicators can comprehensively reflect the performance of the model in classification or similar document retrieval tasks; S7c: Comparative model settings: In order to verify the effectiveness of the fine-tuned RoBERTa model, several baseline models were selected for comparison, including the original RoBERTa model without fine-tuning, BERT, and the combination of traditional TF-IDF plus machine learning classifiers (such as SVM) as benchmarks for comparative analysis. Such comparisons can intuitively show the improvement of model performance during the fine-tuning process; S7d: Experimental environment: All experiments are conducted in a high-performance computing environment with NVIDIA Tesla GPU support to ensure high efficiency of the training and testing process; S8: Experimental results: S8a: Document classification task: The fine-tuned RoBERTa model achieved an accuracy of 92% on the test set, compared to 85% for the original RoBERTa model without fine-tuning, 87% for the traditional BERT model and 83% for the combination of TF-IDF plus SVM classifier, indicating that the fine-tuning process significantly improved the performance of the RoBERTa model on the specific task; S8b: Similar document retrieval task: Using cosine similarity as the similarity scoring criterion, the fine-tuned RoBERTa model also outperforms the comparison group in the retrieval task. Specifically, the retrieval system based on the fine-tuned RoBERTa model achieves 90% Top-5 accuracy, while the original RoBERTa model and BERT model are 82% and 85% respectively, and the TF-IDF plus SVM combination method is 80%, which further confirms the importance of fine-tuning in improving the model's deep semantic understanding of documents and retrieval performance; S8c: Training and inference time: In terms of training efficiency, the process of fine-tuning the RoBERTa model is much shorter than training the model from scratch. Specifically, the entire fine-tuning process takes an average of about 2 hours in an experimental environment, while training a model of similar size from scratch may take more than 12 hours.

[0007] As a further technical solution of the present invention, in S3a, in order to ensure the quality and availability of data, only documents with complete content and unified format are selected; considering that the goal of the knowledge base is to promote the sharing and reuse of knowledge, special attention is paid to innovative achievement documents with universal applicability and transformation potential.

[0008] As a further technical solution of the present invention, in S3b, non-text format documents include PDF, Word, etc., and formatting elements include pictures, tables, special formats, etc.

[0009] As a further technical solution of the present invention, in S3c, the irrelevant information includes headers, footers, chart descriptions, references, etc.

[0010] As a further technical solution of the present invention, in S3d, the Chinese word segmentation tool based on deep learning adopts the BERT word segmenter.

[0011] As a further technical solution of the present invention, in S4c, the specific parameter setting and adjustment strategy are: Learning rate: Choose a relatively small learning rate (such as 5e-5 or smaller) to prevent excessive perturbations to the pre-trained model parameters during fine-tuning; Batch size: According to the available computing resources, the batch size should be adjusted appropriately to ensure the model training efficiency while avoiding training problems caused by resource limitations.

[0012] Training cycle: Set reasonable training cycles (Epochs). Too few training cycles may lead to underfitting of the model, while too many training cycles may lead to overfitting. The training cycle can be adjusted dynamically by monitoring the model performance on the validation set.

[0013] As a further technical solution of the present invention, in S5, the model fine-tuning process is usually monitored on a validation set to determine when the model reaches optimal performance or begins to overfit.

[0014] As a further technical solution of the present invention, in S6, the entity type of the document includes project name, key technology, effect index, etc.

[0015] As a further technical solution of the present invention, in S7a, the training set is used for fine-tuning the model, the validation set is used for adjusting model parameters and applying the early stopping method, and the test set is used for final performance evaluation.

[0016] As a further technical solution of the present invention, in S8c, the response speed of the fine-tuned model during reasoning also meets the requirements of real-time retrieval, and the average response time is within 100 milliseconds, which is crucial for building a practical knowledge base system.

[0017] The beneficial effects of the present invention are: by fine-tuning the RoBERTa model, efficient document classification and similar document retrieval are successfully achieved in the task of building a knowledge base of tobacco management innovation achievements, demonstrating the application potential of natural language processing technology based on deep learning in the field of professional knowledge base construction; the main advantages of the present invention include: 1. Powerful contextual understanding capabilities: Compared with traditional text processing methods, RoBERTa-based methods can better capture and understand the complex context and subtle semantic differences in documents, which is crucial for accurate classification and effective retrieval and management of innovative documents.

[0018] 2. Flexible fine-tuning capabilities: The strategy of fine-tuning the pre-trained model provides great flexibility and efficiency for optimizing the model on specific tasks. By fine-tuning the specific data of tobacco, the RoBERTa model can more accurately adapt to and process the document characteristics of management innovation results, rather than relying on a general language model.

[0019] 3. Efficient data processing: In terms of processing large-scale document datasets, the method of fine-tuning the RoBERTa model shows efficient data processing capabilities, which is especially important in the task of building a knowledge base with a large amount of documents and frequent updates.

[0020] 4. Improved user experience: The knowledge base system based on the fine-tuned RoBERTa model has been highly praised by users in terms of the relevance of retrieval results and the system response speed. Providing fast and accurate retrieval results directly affects the user experience and the practical value of the knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the method for automatic classification and storage of tobacco industry documents using deep learning proposed by the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0023] Please refer to the attached Figure 1 , the automatic classification and storage method of tobacco industry documents using deep learning includes the following specific steps: S1: Model selection: Considering the large amount of Chinese documents in the tobacco industry and the need to accurately understand complex management knowledge, RoBERTa was selected as the basis for the pre-training model. RoBERTa's efficient information processing mechanism and excellent understanding of Chinese corpus make it an ideal choice for building a knowledge base and realizing intelligent search functions; S2: Model pre-training process: S2a: MaskedLanguageModel (MLM) stage: The model randomly masks part of the vocabulary in the input text, and then predicts these masked words, in this way learning the deep semantic relationship of the language; S2b: Dynamic mask generation stage: Unlike BERT, RoBERTa dynamically changes the mask vocabulary at each data pass, which increases the difficulty and diversity of model training; RoBERTa is a pre-trained language model optimized based on the BERT architecture. By training for a longer time on a larger dataset and adopting a dynamic masking mechanism, RoBERTa has demonstrated performance that exceeds the original BERT model on multiple natural language processing tasks. In addition, RoBERTa abandons the NextSentencePrediction (NSP) task used by BERT in the pre-training phase and focuses on more refined language model training. These improvements give the RoBERTa model a significant advantage in understanding context and grasping language nuances.

[0024] RoBERTa inherits the Transformer architecture of BERT, which is based on the self-attention mechanism and can capture long-range dependencies in the text. Each Transformer layer contains multi-head self-attention and a fully connected feed-forward network, and this design enables the model to process text data in parallel, effectively improving the training efficiency.

[0025] S3: Data Preparation and Preprocessing: S3a: Data Collection: Collect industry documents from the internal database of tobacco companies; S3b: Document Format Conversion: Use automated scripts to convert all non-text format documents into plain text format, removing formatting elements for subsequent processing; S3c: Text Cleaning: Clean the converted text content, removing irrelevant information and also剔除 or replacing special characters and unstructured information in the text; S3d: Word Segmentation: Considering the characteristics of Chinese documents, use a deep learning-based Chinese word segmentation tool for word segmentation to ensure that the text is correctly segmented into words or phrases, and standardize all text to lowercase (for English parts) to unify the data format; S3e: Stop Word Removal: According to a predefined stop word list, remove common stop words in the text, such as "的", "是", "在", etc. These words, although frequently used, contribute limitedly to understanding the document meaning; S3f: Data Augmentation: To avoid data bias during model training, adopt a data augmentation strategy. For example, generate new text instances through methods such as synonym replacement and sentence restructuring, which can help the model learn more generalized language features and improve performance on different types of documents; S4: Model Fine-tuning Strategy: S4a: Define Task-specific Model Architecture: For knowledge base construction, it usually involves tasks such as document classification or similar document retrieval. Therefore, add one or more fully connected layers on top of the RoBERTa language model as the output layer to adapt to these tasks; S4b: Select Appropriate Loss Function and Optimizer: According to the different tasks, select an appropriate loss function to guide model training; for classification tasks, use the cross-entropy loss function; for similarity matching tasks, use cosine similarity as the loss function; at the same time, select an effective optimizer (such as AdamW) to adjust model parameters to achieve better training effects; S4c: Parameter Setting and Adjustment Strategy: Set and adjust the learning rate, batch size, and number of training epochs; S5: Model fine-tuning process: The preprocessed document data is input into the RoBERTa model, and the loss is calculated based on the output of the task-specific layer, and then the model parameters are adjusted through the back-propagation algorithm; S5a: Data batching: Since document datasets can be quite large, it is usually necessary to input data into the model in batches, which can not only improve memory utilization but also help the generalization ability of the model; S5b: Early Stopping: To avoid overfitting, you can terminate the training early when the performance of the validation set no longer improves. This method can help us find a balance between improving model performance and preventing overfitting. S6: Entity recognition and labeling: After the model is fine-tuned, the fine-tuned RoBERTa model is used to perform named entity recognition tasks. By defining a set of entity types for management innovation results documents, the model can automatically identify key information in the document and mark it as the corresponding entity, providing support for automatic labeling of the document; S7: Performance evaluation: S7a: Dataset construction: From the management innovation achievement documents collected by tobacco companies, 80% are randomly selected as training sets, 10% as validation sets, and the remaining 10% as test sets to ensure that the dataset covers a wide range of management innovation topics and content types; S7b: Performance evaluation indicators: In order to comprehensively evaluate the performance of the model, several common evaluation indicators were selected, including precision, recall, F1 score, and accuracy of the model on the test set. These indicators can comprehensively reflect the performance of the model in classification or similar document retrieval tasks; In order to comprehensively evaluate the application effect of the fine-tuned RoBERTa model in the knowledge base of tobacco companies' management innovation achievements, a series of detailed experiments were designed, which included not only quantitative performance evaluation but also qualitative case analysis to reveal the application potential and limitations of the model in actual knowledge base scenarios.

[0026] S7c: Comparative model settings: In order to verify the effectiveness of the fine-tuned RoBERTa model, several baseline models were selected for comparison, including the original RoBERTa model without fine-tuning, BERT, and the combination of traditional TF-IDF plus machine learning classifiers (such as SVM) as benchmarks for comparative analysis. Such comparisons can intuitively show the improvement of model performance during the fine-tuning process; S7d: Experimental environment: All experiments are conducted in a high-performance computing environment with NVIDIA Tesla GPU support to ensure high efficiency of the training and testing process; S8: Experimental results: S8a: Document classification task: The fine-tuned RoBERTa model achieved an accuracy of 92% on the test set, compared to 85% for the original RoBERTa model without fine-tuning, 87% for the traditional BERT model and 83% for the combination of TF-IDF plus SVM classifier, indicating that the fine-tuning process significantly improved the performance of the RoBERTa model on the specific task; S8b: Similar document retrieval task: Using cosine similarity as the similarity scoring criterion, the fine-tuned RoBERTa model also outperforms the comparison group in the retrieval task. Specifically, the retrieval system based on the fine-tuned RoBERTa model achieves 90% Top-5 accuracy, while the original RoBERTa model and BERT model are 82% and 85% respectively, and the TF-IDF plus SVM combination method is 80%, which further confirms the importance of fine-tuning in improving the model's deep semantic understanding of documents and retrieval performance; S8c: Training and inference time: In terms of training efficiency, the process of fine-tuning the RoBERTa model is much shorter than training the model from scratch. Specifically, the entire fine-tuning process takes an average of about 2 hours in an experimental environment, while training a model of similar size from scratch may take more than 12 hours.

[0027] In a preferred embodiment, in S3a, to ensure the quality and availability of data, only documents with complete content and uniform format are selected; considering that the goal of the knowledge base is to promote the sharing and reuse of knowledge, special attention is paid to innovative achievement documents with universal applicability and transformation potential.

[0028] In a preferred embodiment, in S3b, non-text format documents include PDF, Word, etc., and formatting elements include pictures, tables, special formats, etc.

[0029] In a preferred embodiment, in S3c, irrelevant information includes headers, footers, chart descriptions, references, etc.

[0030] In a preferred embodiment, in S3d, the Chinese word segmentation tool based on deep learning adopts the BERT word segmenter.

[0031] In a preferred embodiment, in S4c, the specific parameter setting and adjustment strategy is: Learning rate: Choose a relatively small learning rate (such as 5e-5 or smaller) to prevent excessive perturbations to the pre-trained model parameters during fine-tuning; Batch size: According to the available computing resources, the batch size should be adjusted appropriately to ensure the model training efficiency while avoiding training problems caused by resource limitations.

[0032] Training cycle: Set reasonable training cycles (Epochs). Too few training cycles may lead to underfitting of the model, while too many training cycles may lead to overfitting. The training cycle can be adjusted dynamically by monitoring the model performance on the validation set.

[0033] In a preferred embodiment, in S5, the model fine-tuning process is usually monitored on the validation set to determine when the model reaches optimal performance or begins to overfit.

[0034] In a preferred implementation, in S6, the entity types of the document include project name, key technology, effect indicators, etc.

[0035] In a preferred embodiment, in S7a, the training set is used for fine-tuning the model, the validation set is used for adjusting model parameters and applying the early stopping method, and the test set is used for the final performance evaluation.

[0036] In a preferred embodiment, in S8c, the response speed of the fine-tuned model during reasoning also meets the requirements of real-time retrieval, and the average response time is within 100 milliseconds, which is crucial for building a practical knowledge base system.

[0037] 1. Analysis of the reasons for performance improvement Enhanced contextual understanding capabilities: The RoBERTa model has learned rich language features and deep semantic information through large-scale pre-training. The fine-tuning process further adapts the model to the specific context of managing innovative achievement documents. This enhanced contextual understanding capability is one of the main reasons for the performance improvement.

[0038] Data-specific adaptation: By fine-tuning on the management innovation documents of tobacco companies, the RoBERTa model is able to better adapt to the specific characteristics of these documents, such as professional terminology, organization-specific expressions, etc. This data-specific adaptation significantly improves the model's accuracy in document classification and similar document retrieval tasks.

[0039] Effectiveness of model optimization strategy: Choosing the right loss function and optimizer, as well as properly setting the learning rate and training cycle, are crucial to the success of the fine-tuning process. These optimization strategies help the model retain the knowledge acquired during the pre-training phase while learning new tasks, thus avoiding overfitting and ensuring improved model performance.

[0040] 2. Significance of experimental findings Balance between model generality and specificity: By properly fine-tuning the pre-trained model, it is possible to effectively retain the generality of the model while enhancing its performance on specific tasks, which provides a feasible method for using pre-trained models to solve problems in specific fields.

[0041] New approaches to knowledge base construction: Using advanced natural language processing technology to build a knowledge base, especially in professional fields such as managing innovative achievements, provides a new approach to the accumulation, sharing and utilization of knowledge within an organization, which is of great significance for improving the organization's innovation capabilities and decision-making efficiency.

[0042] Improved user experience: The user satisfaction survey results reflect the efficiency of the knowledge base system based on the fine-tuned RoBERTa model in practical applications and the significant improvement in user experience, which emphasizes the need to focus on the actual needs and usage experience of end users when developing and optimizing knowledge base systems.

[0043] 3. Application prospects This method successfully uses the fine-tuned RoBERTa model to improve the construction and query functions of the knowledge base of tobacco companies' management innovation results, which not only provides a powerful knowledge management tool for tobacco companies, but also provides new ideas and methods for the construction of knowledge bases in other industries and fields. The following are several potential application prospects of this method: Cross-industry knowledge base construction: The method of fine-tuning the pre-trained language model can be widely used in the construction of knowledge bases in other industries, such as medicine, law, education and other fields. Each field has its own unique expertise and language expression. Through targeted fine-tuning, the model can better adapt to the needs of specific fields and improve the quality and practicality of the knowledge base.

[0044] Intelligent question-answering system: Based on the text understanding ability of deep learning, the fine-tuned model can serve as the core of the intelligent question-answering system to provide accurate and relevant answers; this application has great potential in scenarios such as customer service, online education, and medical consultation, and can improve user experience and service efficiency.

[0045] Personalized recommendation system: Applying the fine-tuned RoBERTa model to the recommendation system can analyze the deep semantic information of the content based on the user's historical behavior and preferences, thereby providing more personalized and accurate recommendations; this has important commercial value for content platforms, e-commerce websites, etc.

[0046] Language and sentiment analysis: Businesses and organizations can use fine-tuned models to perform language and sentiment analysis, such as analyzing customer feedback, market research reports, or public sentiment on social media; this helps companies better understand customer needs and market trends, and provide data support for decision-making.

[0047] Promote the dynamic development of knowledge: In addition, applying deep learning technology to knowledge base construction can also promote the dynamic development of knowledge; as new knowledge is constantly generated, the knowledge base needs to be updated and expanded regularly; through automated text processing and intelligent classification, the newly added knowledge content can be effectively managed and integrated to ensure the continuous growth and timeliness of the knowledge base, thereby supporting the organization's continuous learning and innovation.

[0048] From the above description, it can be seen that the above-mentioned embodiments of the present invention achieve the following technical effects: by fine-tuning the RoBERTa pre-training model and combining it with the BERT vectorization scheme, a knowledge base of management innovation results is successfully constructed for the tobacco company; this knowledge base not only supports efficient document classification and similar document retrieval, but also proves its practicality and effectiveness in improving knowledge sharing and utilization efficiency through user satisfaction surveys; the results of this method demonstrate the great potential of deep learning technology in processing professional knowledge base construction tasks, especially in understanding complex management innovation documents and providing accurate information retrieval.

[0049] This method emphasizes the importance of fine-tuning pre-trained models in specific domains, a process that significantly improves the performance of the model on specific tasks. Through careful experimental design and results analysis, the superior performance of the fine-tuned RoBERTa model in document classification and similar document retrieval tasks is verified. At the same time, through comparison with other models, the effectiveness of this method is further confirmed.

[0050] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention (including the claims) is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0051] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatic classification and storage of tobacco industry documents using deep learning, characterized in that: It includes the following specific steps: S1: Model selection: Considering the large number of Chinese documents in the tobacco industry and the need to accurately understand complex management knowledge, RoBERTa is selected as the basis for the pre-trained model. RoBERTa's efficient information processing mechanism and excellent understanding ability of Chinese corpora make it an ideal choice for building a knowledge base and implementing intelligent search functions; S2: Model pre-training process: S2a: Masked Language Model (MLM) stage: The model randomly masks a part of the vocabulary in the input text and then predicts these masked words, learning the deep semantic relationships of the language in this way; S2b: Dynamic mask generation stage: Different from BERT, RoBERTa dynamically changes the masked words each time the data is passed, increasing the difficulty and diversity of model training; S3: Data preparation and preprocessing: S3a: Data collection: Collect industry documents from the internal database of the tobacco company; S3b: Document format conversion: Use automated scripts to convert all non-text format documents into plain text format, removing formatting elements for subsequent processing; S3c: Text cleaning: Clean the converted text content, remove irrelevant information, and at the same time remove or replace special characters and unstructured information in the text; S3d: Word segmentation processing: Considering the characteristics of Chinese documents, use a deep learning-based Chinese word segmentation tool for word segmentation processing to ensure that the text is correctly segmented into words or phrases, and standardize all text to lowercase (for English parts) to unify the data format; S3e: Stop word removal: According to the predefined stop word list, remove common stop words in the text, such as "de", "shi", "zai", etc. Although these words are frequently used, they contribute limitedly to understanding the meaning of the document; S3f: Data augmentation: To avoid data bias during model training, adopt a data augmentation strategy. For example, generate new text instances through methods such as synonym replacement and sentence restructuring, which can help the model learn more generalized language features and improve performance on different types of documents; S4: Model fine-tuning strategy: S4a: Define a task-specific model architecture: For knowledge base construction, tasks such as document classification or similar document retrieval are usually involved. Therefore, add one or more fully connected layers on top of the RoBERTa language model as the output layer to adapt to these tasks; S4b: Select appropriate loss functions and optimizers: According to the different tasks, select appropriate loss functions to guide model training; for classification tasks, use the cross-entropy loss function; for similarity matching tasks, use cosine similarity as the loss function; at the same time, select an effective optimizer (such as AdamW) to adjust the model parameters to achieve better training effects; S4c: Parameter setting and adjustment strategy: Set and adjust the learning rate, batch size, and number of training epochs; S5: Model fine-tuning process: Input the preprocessed document data into the RoBERTa model, calculate the loss based on the output of the task-specific layer, and then adjust the model parameters through the backpropagation algorithm; S5a: Data batching: Since document datasets can be quite large, it is usually necessary to input data into the model in batches, which can not only improve memory utilization but also help the generalization ability of the model; S5b: Early Stopping: To avoid overfitting, you can terminate the training early when the performance of the validation set no longer improves. This method can help us find a balance between improving model performance and preventing overfitting. S6: Entity recognition and labeling: After the model is fine-tuned, the fine-tuned RoBERTa model is used to perform named entity recognition tasks. By defining a set of entity types for management innovation results documents, the model can automatically identify key information in the document and mark it as the corresponding entity, providing support for automatic labeling of the document; S7: Performance evaluation: S7a: Dataset construction: From the management innovation achievement documents collected by tobacco companies, 80% are randomly selected as training sets, 10% as validation sets, and the remaining 10% as test sets to ensure that the dataset covers a wide range of management innovation topics and content types; S7b: Performance evaluation indicators: In order to comprehensively evaluate the performance of the model, several common evaluation indicators were selected, including precision, recall, F1 score, and accuracy of the model on the test set. These indicators can comprehensively reflect the performance of the model in classification or similar document retrieval tasks; S7c: Comparative model settings: In order to verify the effectiveness of the fine-tuned RoBERTa model, several baseline models were selected for comparison, including the original RoBERTa model without fine-tuning, BERT, and the combination of traditional TF-IDF plus machine learning classifiers (such as SVM) as benchmarks for comparative analysis. Such comparisons can intuitively show the improvement of model performance during the fine-tuning process; S7d: Experimental environment: All experiments are conducted in a high-performance computing environment with NVIDIA Tesla GPU support to ensure high efficiency of the training and testing process; S8: Experimental results: S8a: Document classification task: The fine-tuned RoBERTa model achieved an accuracy of 92% on the test set, compared to 85% for the original RoBERTa model without fine-tuning, 87% for the traditional BERT model and 83% for the combination of TF-IDF plus SVM classifier, indicating that the fine-tuning process significantly improved the performance of the RoBERTa model on the specific task; S8b: Similar document retrieval task: Using cosine similarity as the similarity scoring criterion, the fine-tuned RoBERTa model also outperforms the comparison group in the retrieval task. Specifically, the retrieval system based on the fine-tuned RoBERTa model achieves 90% Top-5 accuracy, while the original RoBERTa model and BERT model are 82% and 85% respectively, and the TF-IDF plus SVM combination method is 80%, which further confirms the importance of fine-tuning in improving the model's deep semantic understanding of documents and retrieval performance; S8c: Training and inference time: In terms of training efficiency, the process of fine-tuning the RoBERTa model is much shorter than training the model from scratch. Specifically, the entire fine-tuning process takes an average of about 2 hours in an experimental environment, while training a model of similar size from scratch may take more than 12 hours.

2. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1 is characterized in that: In S3a, in order to ensure the quality and availability of data, only documents with complete content and uniform format are selected; considering that the goal of the knowledge base is to promote the sharing and reuse of knowledge, special attention is paid to innovative achievement documents with universal applicability and transformation potential.

3. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1 is characterized in that: In the S3b, non-text format documents include PDF, Word, etc., and formatting elements include pictures, tables, special formats, etc.

4. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1 is characterized in that: In S3c, irrelevant information includes headers, footers, chart descriptions, references, etc.

5. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1 is characterized in that: In the S3d, the Chinese word segmentation tool based on deep learning adopts the BERT word segmenter.

6. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1 is characterized in that: In S4c, the specific parameter settings and adjustment strategies are as follows: Learning rate: Choose a relatively small learning rate (such as 5e-5 or smaller) to prevent excessive perturbations to the pre-trained model parameters during fine-tuning; Batch size: According to the available computing resources, the batch size (BatchSize) should be adjusted appropriately to ensure the model training efficiency while avoiding training problems caused by resource limitations; Training cycle: Set reasonable training cycles (Epochs). Too few training cycles may lead to underfitting of the model, while too many training cycles may lead to overfitting. The training cycle can be adjusted dynamically by monitoring the model performance on the validation set.

7. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1 is characterized in that: In S5, the model fine-tuning process is usually monitored on the validation set to determine when the model reaches optimal performance or begins to overfit.

8. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1 is characterized in that: In S6, the entity types of the document include project name, key technology, effect indicators, etc.

9. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1, characterized in that: In S7a, the training set is used for fine-tuning the model, the validation set is used for adjusting model parameters and applying the early stopping method, and the test set is used for the final performance evaluation.

10. The method for automatic classification and storage of tobacco industry documents using deep learning according to claim 1, characterized in that: In S8c, the response speed of the fine-tuned model during reasoning also meets the requirements of real-time retrieval, and the average response time is within 100 milliseconds, which is crucial for building a practical knowledge base system.

Citation Information

Patent Citations

  • Aspect level sentiment analysis method and device based on transfer learning

    CN114912423A

Cited By

  • Agricultural subject classification method for papers

    CN119807425A

  • File management system and file classification coding method thereof

    CN120218543A

  • Method and system for constructing manuscript review agent embedded in gradient evaluation system

    CN120597933A

  • Dynamic self-adaptive operation and maintenance data intelligent prediction system, method and server

    CN120806287A