A method for constructing a forestry pre-trained language model based on continuous learning

Abstract

A method for constructing a forestry pre-trained language model based on continuous learning belongs to the fields of artificial intelligence and computer program technology. Construction of a forestry corpus: Use web crawlers, database queries, and information extraction methods to collect text data on three topics, namely terms related to forestry, laws and regulations, and literature, from the China Forestry Information Network and forestry-related Peking University core journals. After data cleaning and preprocessing such as removing special symbols, duplicate values, and word segmentation, three forestry corpora for pre-training are obtained. Using these three forestry corpora with different topics, continuous pre-training is performed on the pre-trained language model BERT in the general domain based on the continuous learning method to enhance the knowledge of the pre-trained language model in the general domain within the forestry field, and finally a forestry pre-trained language model ForestryBERT that can continuously learn is formed.

Description

Technical Field

[0001] The present invention belongs to the technical fields of artificial intelligence and computer programs, and relates to a method for constructing a forestry pre-trained language model based on continuous learning. Background Art

[0002] With the continuous development of forestry informatization, the forestry text data containing a large amount of valuable information (such as scientific publications, reports, etc.) has shown an explosive growth. Although text has been emphasized in many fields, it has rarely been effectively utilized in forestry research. How to efficiently and quickly mine hidden patterns and rules from large-scale unlabeled forestry text data remains a major challenge.

[0003] Some studies have begun to use natural language processing (NLP) techniques to process and analyze forestry text. For example, extracting keywords and text summaries from text based on unsupervised algorithms such as TextRank, or classifying forestry text based on machine learning and deep learning. However, the relevant research is still limited, and usually relies on traditional NLP paradigms, as well as designing specialized models to complete specific tasks, and the generality of the models is poor, making it difficult to meet the needs of efficiently and quickly processing text information.

[0004] Pre-trained language models have shown powerful capabilities in processing a large amount of unlabeled text. By performing self-supervised pre-training on a large-scale unlabeled corpus to obtain general knowledge independent of tasks, and then using a small-scale task-specific labeled dataset to fine-tune the pre-trained model to complete various downstream tasks, such as text classification, question answering, etc. This brings new inspiration to large-scale forestry text processing. Although existing forestry research has tried to use pre-trained language models to process forestry text, the models are only used to generate word vectors, without giving full play to the advantages of their pre-training + fine-tuning paradigm. At present, pre-trained language models have been successfully applied to multiple fields. For the forestry field that needs to effectively mine a large amount of text to realize information value-added, applying pre-trained language models is equally crucial.

[0005] Pre-trained language models are usually trained on general-domain corpora. For example, BERT is based on Wikipedia and BookCorpus. If these models are directly used for downstream tasks in specific domains, their performance usually degrades. Therefore, to enhance domain-specific knowledge and adapt to downstream tasks in specific domains, existing research usually uses two methods: training from scratch and continued pre-training to construct domain-specific pre-trained language models. Especially domain-specific pre-trained language models based on BERT as the base model, such as Med-BERT in the medical field, AgriBERT in the agricultural field, BioBERT in the biomedical field, and FinBERT in the financial field. However, both of these methods perform one-time pre-training using a single target corpus, which is inconsistent with the continuous adaptation and knowledge accumulation process in the real world. Summary of the Invention

[0006] Aiming at the problems of the existing technology, the present invention provides a method for constructing a forestry pre-trained language model based on continuous learning.

[0007] A method for constructing a forestry pre-trained language model based on continuous learning includes the following steps:

[0008] Step 1: Construction of a forestry corpus:

[0009] Use web crawlers, database queries, and information extraction methods to collect text data on three topics: forestry-related terms, laws and regulations, and literature from the China Forestry Information Network and forestry-related core journals of Peking University. After removing special symbols, duplicate values, and performing word segmentation data cleaning and preprocessing, three forestry corpora for pre-training are obtained.

[0010] Step 2: Continuous pre-training of the model:

[0011] Use three forestry corpora with different topics to perform continuous pre-training on the general-domain pre-trained language model BERT based on the continuous learning method, enhance the knowledge of the general-domain pre-trained language model in the forestry field, and finally form a forestry pre-trained language model ForestryBERT that can continuously learn.

[0012] Step 3: Fine-tuning of the model:

[0013] Step 3.1: Construction of a forestry annotation dataset.

[0014] Step 3.1.1 Forestry text classification dataset.

[0015] The original data crawled from the China Forestry Information Network has classification tags, but each topic has different classification criteria and quantities. Samples with similar tags in each topic are merged, and then samples are randomly selected from each dataset at a ratio of 1% to 10%, restricting the categories with a larger quantity. Finally, three forestry text datasets with different topics are obtained.

[0016] Step 3.1.2 Forestry extractive Q&A dataset.

[0017] Use the labeled Chinese reading comprehension dataset to fine-tune the pre-trained language model to build a model capable of generating Q&A pairs. Randomly select texts from the three forestry corpora, and use this model to generate corresponding answers and questions for each context. After manual verification, a forestry extractive Q&A dataset consisting of 2280 samples is constructed.

[0018] Step 3.2, Fine-tuning: Fine-tune the three models generated during the continuous pre-training process according to the two constructed labeled datasets to evaluate the performance of the forestry pre-trained language model and the effectiveness of the continuous learning method.

[0019] The specific process of the continuous pre-training method in Step 2 is as follows:

[0020] [[ID=1;5]]Construct the three topic corpora as a sample set stream: {C1, C2, C3}, which are the forestry, laws and regulations, and literature topics respectively. Based on BERT as the base model and LERT as the initial weight (Weight LERT ), first calculate the importance scores in the general domain according to the important units in the multi-head attention layer, intermediate layer, and output layer of the Transformer model based on the self-attention mechanism Then, based on this score, perform domain training on the C1 topic to obtain the Weight1 model with forestry term knowledge; at the same time, calculate the importance score of the C1 topic As the old knowledge of the C2 topic, and then in the way of the maximum value of the corresponding elements To accumulate the importance scores to obtain the importance scores of the first two topics Then, based on this score, perform domain training on the C2 topic to obtain the Weight2 model with forestry term and laws and regulations knowledge; in the same way, obtain the importance score of the C2 topic and the cumulative importance scores of the first three topics And based on this score, domain training is carried out on the C3 theme to obtain the Weight3 model with all the knowledge of these three forestry themes. These three models have learned different theme knowledge by accumulating importance. Weight1 has learned the C1 theme, Weight2 has learned the C1 and C2 themes, and Weight3 (i.e., ForestryBERT) has learned the three themes. Then, the models generated in each stage are evaluated by fine-tuning the downstream tasks of the corresponding themes.

[0021] The domain training of the theme includes backward propagation and forward propagation. During backward propagation, the importance score is used to update the model weights while retaining the previous knowledge. During forward propagation, for the convenience of knowledge transfer, the obtained weights are divided into two parts. One part is the product of the accumulated importance score and the elements of the output of each layer, which only contains the previous knowledge, and the other part is the complete knowledge, which contains the knowledge of the previous and current domains.

[0022] In step 3.1, in the construction of the forestry annotation dataset, there is step 3.1.2, the forestry extractive Q&A dataset. The construction of the Q&A pair generation model is specifically described as follows:

[0023] Based on the pre-trained language model mT5, using the Chinese reading comprehension data SQuAD as the training sample, mT5 is fine-tuned based on the two-stage tasks of answer span extraction and question generation. Subsequently, the unannotated forestry corpus containing only the context is input into the fine-tuned question-answer generation model to generate context-based questions and answer pairs to construct a synthetic triple dataset of context, answer, and question for the fine-tuning of ForestryBERT.

[0024] In step 3.2, in the fine-tuning, there is step 3.2.1, the text classification task: Text classification is an automatic process of classifying text data according to predefined categories, focusing on the single-label multi-class classification task. According to different classification objectives, it is divided into two sub-tasks: regarding samples of the same theme as different classes and further subdividing each theme; regarding samples of the same theme as the same class and directly classifying according to the three themes. The first sub-task fine-tunes the three models on the text classification datasets of different themes to observe whether the models generated in the pre-training stage can effectively retain the old knowledge while learning new knowledge. The second sub-task only fine-tunes the ForestryBERT model on a forestry text classification dataset containing three themes to observe its performance when completing forestry tasks.

[0025] In the fine-tuning of Step 3.2, there is Step 3.2.2, Extractive Question Answering Task Classification: To comprehensively evaluate whether ForestryBERT effectively learns forestry expertise, the constructed forestry extractive question answering dataset is used to further fine-tune the model to complete the extractive question answering task. The model automatically extracts relevant information from the given text and question and provides answers.

[0026] The advantages of this invention are that a pre-trained language model based on Transformer is constructed. This model can understand forestry semantics, can continuously adapt to new scenarios, and has the ability of scalability and sustainable learning. For the first time, this invention constructs a pre-trained language model for a specific domain through a continuous learning method rather than simple continued pre-training, which conforms to the situation of humans acquiring knowledge in real scenarios. ForestryBERT outperforms multiple pre-trained language models in general domains in two forestry tasks. Compared with the continued pre-trained language model, it can retain old knowledge while learning new knowledge.

[0027] ForestryBERT constructed by the method proposed in this invention has the capabilities of sustainability, scalability, and continuous learning, and can efficiently complete various forestry downstream tasks. By enhancing the forestry knowledge of the pre-trained language model in general domains through continuous pre-training, the accuracy of the general model in completing forestry tasks can be improved. In addition, the continuous learning method adopted in the continuous pre-training stage can effectively promote new knowledge acquisition and alleviate catastrophic forgetting, which means that the model can, based on the knowledge learned in the past, continuously perform pre-training according to the need by adding new corpora, indicating the feasibility of the continuous learning method in constructing a pre-trained language model for a specific domain. By maximizing the value of forestry text information, this invention brings new insights to large-scale text processing in forestry and provides a valuable example for constructing pre-trained language models in other industries. Brief Description of the Drawings

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. As shown in the figure:

[0029] Figure 1 It is the program flow chart of the present invention.

[0030] Figure 2 It is the construction diagram of the question answer generation model of the present invention.

[0031] Figure 3This is one of the fine-tuning diagrams of different models of the present invention for text classification of different themes.

[0032] Figure 4 This is the second fine-tuning diagram of different models of the present invention for text classification of different themes. Detailed implementation manners

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] Embodiment 1: As Figure 1 , Figure 2 , Figure 3 and Figure 4 shown, a method for constructing a forestry pre-trained language model based on continuous learning aims to solve the problems of low utilization rate of current forestry texts, poor generality of traditional text processing models, and inability of existing pre-trained language models in specific fields to continuously learn. The method for constructing a forestry pre-trained language model based on continuous learning proposed by the present invention. The model constructed by this method can not only understand forestry semantic information but also has the ability of continuous learning, which helps to improve the work efficiency of forestry workers and provide support for intelligent information services in the forestry field.

[0035] A method for constructing a forestry pre-trained language model based on continuous learning includes the steps of constructing a forestry corpus, continuously pre-training the model, and fine-tuning the model. Its overall technical process is as Figure 1 shown.

[0036] It includes the following steps:

[0037] Step 1: Construction of the forestry corpus:

[0038] Collect text data on three themes of terms, laws and regulations, and literature related to forestry from channels such as the China Forestry Information Network and forestry-related core journals of Peking University by using technologies such as web crawlers, database queries, and information extraction. After data cleaning and preprocessing such as removing special symbols, duplicate values, and word segmentation, three forestry corpora for pre-training are obtained.

[0039] Step 2: Continuous pre-training of the model:

[0040] Using three forestry corpora with different themes, based on the continuous learning method, continuously pre-train the pre-trained language model BERT in the general domain to enhance the knowledge of the pre-trained language model in the general domain within the forestry field, and finally construct a forestry pre-trained language model ForestryBERT that can continuously learn. The specific process of the continuous pre-training method is as follows:

[0041] The three theme corpora constructed in this invention form a sample set stream: {C1, C2, C3}. Taking BERT as the base model and LERT as the initial weight (Weight LERT ), first calculate the importance scores in the general domain according to the important units in the multi-head attention layer, intermediate layer, and output layer of the Transformer model based on the self-attention mechanism Then, based on this score, perform domain training on the C1 theme to obtain the Weight1 model with forestry term knowledge; at the same time, calculate the importance score of the C1 theme As the old knowledge of the C2 theme, and then in the way of the maximum value of the corresponding elements Accumulate the importance scores to obtain the importance scores of the first two themes Then, based on this score, perform domain training on the C2 theme to obtain the Weight2 model with forestry terms and laws and regulations knowledge; in the same way, obtain the importance score of the C2 theme and the cumulative importance scores of the first three themes And based on this score, perform domain training on the C3 theme to obtain the Weight3 model with all the knowledge of these three forestry themes. These three models learn different theme knowledge by accumulating importance. The Weight1 model learns the C1 theme, Weight2 learns the C1 theme and the C2 theme, and the Weight3 (i.e., ForestryBERT) model learns the three themes. Then, evaluate the models generated in each stage by fine-tuning the downstream tasks of the corresponding themes

[0042] The domain training of the theme includes backpropagation and forward propagation. In the backpropagation process, use the importance score to update the model weights while retaining the previous knowledge. In the forward propagation, in order to facilitate knowledge transfer, the obtained weights are divided into two parts. One part is the product of the accumulated importance score and the elements of each layer output, which only contains the previous knowledge. The other part is the complete knowledge, which contains the knowledge of the previous and current domains

[0043] Step 3: Fine-tuning of the model:

[0044] (1) Construction of the forestry annotation dataset

[0045] ① Forestry text classification dataset

[0046] The original data crawled from the China Forestry Information Network has classification tags, but each topic has different classification criteria and quantities. To alleviate the problem of data class imbalance, samples with similar tags in each topic were merged, and then samples were randomly drawn from each dataset at a certain ratio to limit the categories with larger quantities. Finally, three forestry text datasets with different topics were obtained.

[0047] ② Forestry extractive question-answering dataset

[0048] A pre-trained language model was fine-tuned using a Chinese reading comprehension dataset to build a model that can generate question-answer pairs. Texts were randomly selected from three forestry corpora, and the model was used to generate corresponding answers and questions for each context. After manual verification, a forestry extractive question-answering dataset consisting of 2,280 samples was constructed. The construction of the question-answer pair generation model is described as follows:

[0049] As Figure 2 shown, based on the mT5 model as the base model, a labeled dataset was used as the training sample, and mT5 was fine-tuned based on the two-stage tasks of answer span extraction and question generation. Subsequently, the unlabeled forestry dataset containing only the context was input into the fine-tuned model to generate questions and answers, in order to construct a synthetic triple dataset of context, answer, and question for the fine-tuning of ForestryBERT.

[0050] (2) Fine-tuning

[0051] Three models generated during the continuous pre-training process were fine-tuned according to the two constructed labeled datasets to evaluate the performance of the forestry pre-trained language model and the effectiveness of the continuous learning method.

[0052] ① Text classification task

[0053] Text classification is an automatic process of classifying text data according to predefined categories. The present invention focuses on the single-label multi-category classification task, which is divided into two sub-tasks according to different classification objectives: regarding samples of the same topic as different categories and further subdividing each topic; regarding samples of the same topic as the same category and directly classifying according to the three topics. The first sub-task fine-tunes the three models on text classification datasets of different topics to observe whether the models generated during the pre-training stage can effectively retain old knowledge while learning new knowledge. The second sub-task only fine-tunes the ForestryBERT model on a forestry text classification dataset containing three topics to observe its performance when completing forestry tasks.

[0054] ② Extractive question-answering task classification

[0055] To comprehensively evaluate whether ForestryBERT effectively learns forestry expertise, the constructed forestry extractive Q&A dataset is used to further fine-tune the model to complete the extractive Q&A task. This task requires the model to automatically extract relevant information and provide answers based on the given text and questions to achieve intelligent human-computer interaction, which is more complex than other downstream tasks such as text classification and named entity recognition.

[0056] Example 2: As Figure 1 、 Figure 2 、 Figure 3 and Figure 4 shown, a method for constructing a forestry pre-trained language model based on continual learning includes the following steps:

[0057] Step 1: Establish a dataset:

[0058] (1) Pre-training corpus

[0059] The unannotated corpus includes three topics: forestry terms, forestry laws and regulations, and forestry literature. There are 43,712 items in the forestry terms topic, covering a wide range of forestry professional terms, sourced from the Chinese Forestry Terminology Database on the China Forestry Information Network. There are 10,615 articles in the forestry laws and regulations topic, from the China Forestry Information Network, containing the full texts of forestry policies and laws and regulations in different regions from 2000 to 2023. There are 150,309 articles in the forestry literature topic, obtained from 16 forestry journals in the Peking University Chinese Core Journal Directory (9th Edition) and the master's and doctoral theses of forestry majors in Chinese universities stored on the China Forestry Information Network.

[0060] (2) Forestry text classification dataset

[0061] In the first subtask, the dataset is divided according to the classification labels of each topic, and the ratios for model training, validation, and testing are �:2:2 respectively. In the second subtask, 2,000 samples are selected from each topic to construct a fine-tuning dataset containing three topics, and the division ratio is the same as that of the first subtask. Table 1 shows the classification labels and quantities of different topics in the first task.

[0062] Table 1 Classification of the forestry text classification dataset

[0063]

[0064]

[0065] (3) Extractive Q&A dataset

[0066] According to the different distributions of each topic, the dataset is split at a ratio of 8:2, and finally a training set with 1824 samples and a test set with 456 samples are obtained. Subsequently, the training set is further divided at a ratio of 8:2 for model training and validation. The same test set is used for each test to ensure the consistency of evaluation. The distribution of the dataset is shown in Table 2.

[0067] Table 2 Distribution of the extractive question answering dataset

[0068]

[0069] Step 2. Fine-tuning performance of ForestryBERT on forestry tasks:

[0070] To verify whether the pre-trained language model constructed by the continuous pre-training method based on continuous learning in this invention can improve the performance of forestry downstream tasks, the continuous pre-training method is also used to continuously pre-train three corpora. This process also generates three models that have learned different topic knowledge, and then the same dataset is used for fine-tuning. Table 3 compares the accuracy, F1, and EM scores of three general-domain pre-trained language models, as well as the models constructed by LERT-FT (the final model generated in the continuous pre-training stage) and the method of this invention for completing two tasks. All tests are carried out on the same dataset, and each experiment is repeated five times. ForestryBERT achieved the highest scores in both tasks, being 1.05% higher than the average of the other four models in the text classification task and 2.15% higher than the average of the other four models in the extractive question answering task. In the text classification task, BERT performed the worst, and as the self-supervised pre-training tasks of BERT were improved, the scores of BERT variant models (MacBERT, LERT) gradually increased. In addition, the performance of the continuously pre-trained model LERT-FT also increased slightly. However, in the more complex extractive question answering task, the F1 score of LERT-FT was the lowest, followed by BERT. This may be because LERT-FT did not retain old knowledge during the continuous training process, resulting in poor performance when completing such comprehensive tasks. This reflects that ForestryBERT effectively alleviates catastrophic forgetting and retains old knowledge.

[0071] Table 3 Performance of different models on two forestry tasks

[0072]

[0073] Step 3. Evaluation of the continuous learning effect:

[0074] The present invention performs text classification tasks for three different topics: forestry terms, forestry laws and regulations, and forestry literature. The effect of continuous learning is observed by evaluating the performance of models generated by continuous pre-training using different methods (continuous learning and continued pre-training). During the continuous pre-training process, three models are generated, each corresponding to a new task (the currently learned corpus) and an old task (the previously learned corpus). For example, the literature model (ForestryBERT) is the last model generated during the pre-training process, indicating that the knowledge in the first two corpora has been acquired, and currently only the literature corpus is learned. Therefore, terms and laws and regulations are old tasks, and the literature topic is the new task.

[0075] By fine-tuning each model on new and old tasks, it is observed whether they have acquired new knowledge and prevented catastrophic forgetting, as Figure 3 and Figure 4 shown. After learning the knowledge of the current topic, the models generated by each method perform better than the initial weight LERT (represented by the red line) when completing each corresponding new task. The models generated using the continuous learning method perform the best, especially in the laws and regulations task. For the same classification task, the performance of the models generated at different stages when completing old tasks decreases compared to new tasks. And most of the models with continued pre-training perform even lower than the basic model in old tasks. In addition, compared with the models without using the continuous learning method, the performance degradation of the models using the continuous learning method is much smaller.

[0076] From the results, it can be seen that ForestryBERT constructed by the method proposed in the present invention has the capabilities of sustainability, scalability, and continuous learning, and can efficiently complete various forestry downstream tasks. At the same time, it also proves that the continuous learning method adopted in the continuous pre-training stage can effectively promote the acquisition of new knowledge and alleviate catastrophic forgetting, indicating the feasibility of the continuous learning method in constructing pre-trained language models for specific domains. By maximizing the value of forestry text information, the present invention brings new insights to large-scale text processing in forestry and provides a valuable example for constructing pre-trained language models in other industries.

[0077] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for constructing a forestry pre-training language model based on continuous learning, characterized in that: Contains the following steps: Step 1: Construction of forestry corpus: Web crawlers, database query and information extraction methods were used to collect text data on three topics related to forestry: terminology, laws and regulations, and literature from the China Forestry Information Network and Peking University core journals on forestry. After removing special symbols, repeated values, and word segmentation data cleaning and preprocessing, three forestry corpora for pre-training were obtained. Step 2: Continuous pre-training of the model: Using three forestry corpora with different themes, we continuously pre-trained the general-domain pre-trained language model BERT based on a continuous learning method, enhancing the knowledge of the general-domain pre-trained language model in the forestry field, and ultimately forming a forestry pre-trained language model ForestryBERT that can continuously learn. ForestryBERT is used as the basic model and LERT is used as the initial weight ( ), first calculate the importance score of the general domain based on the importance units in the multi-head attention layer, intermediate layer and output layer of the Transformer model based on the self-attention mechanism , then, based on this score in To gain knowledge of forestry terminology through field training on the subject Model; at the same time, calculate Importance score of the topic , as The old knowledge of the subject is then expressed in the form of the maximum value of the corresponding element ( ) to accumulate the importance scores to get the importance scores of the first two topics , and then based on this score Conduct field training on the subject to acquire knowledge of forestry terminology and laws and regulations Model; in the same way The importance score of the topic and the cumulative importance score of the first three topics , and based on this score in To gain knowledge of all three forestry topics through field training Model, these three models learn different subject knowledge by accumulating importance. Learned theme, The model learned Theme and theme, That is, the ForestryBERT model learns three topics, and then evaluates the model generated at each stage by fine-tuning the downstream tasks of the corresponding topics. Step 3: Fine-tune the model: Step 3.1: Construction of forestry annotation dataset. Step 3.1.1 Forestry text classification dataset, The original data captured from the China Forestry Information Network has classification labels, but each topic has different classification standards and quantities. We merged samples with similar labels in each topic, and then randomly sampled samples at a ratio of 1% to 10% from each dataset, limiting the number of categories. Finally, we obtained three forestry text datasets with different themes. Step 3.1.2 Forestry Extractive Question Answering Dataset, We fine-tune the pre-trained language model using a labeled Chinese reading comprehension dataset to build a model capable of generating question-answer pairs. We then randomly select text from three forestry corpora and use the model to generate corresponding answers and questions for each context. After manual verification, we construct a forestry extractive question-answering dataset consisting of 2,280 samples. Step 3.2, fine-tuning: Fine-tune the three models produced by the continuous pre-training process based on the two constructed labeled datasets to evaluate the performance of the forestry pre-trained language model and the effectiveness of the continuous learning method.

2. The method for constructing a forestry pre-training language model based on continuous learning according to claim 1, characterized in that: The specific process of the continuous pre-training method in step 2 is as follows: The three topic corpora constructed are a sample set stream: , respectively forestry terminology, forestry laws and regulations, and forestry literature topics, The domain training of the subject includes backward propagation and forward propagation. During the backward propagation process, the importance scores are used to update the model weights while retaining the previous knowledge. In the forward propagation, in order to facilitate knowledge transfer, the obtained weights are divided into two parts. One part is the cumulative importance scores multiplied by the elements of the output of each layer, which only contains the previous knowledge, and the other part is the complete knowledge, which includes the knowledge of the previous and current domains.

3. The method for constructing a forestry pre-training language model based on continuous learning according to claim 1, characterized in that: Step 3.1, Construction of the Forestry Annotated Dataset, includes Step 3.1.2, Construction of the Forestry Extractive Question-Answering Dataset. The construction of the question-answer pair generation model is described in detail as follows: Taking the pre-trained language model mT5 as the base model and using the Chinese reading comprehension data SQuAD as training samples, the language model mT5 is fine-tuned based on the two-stage task of answer span extraction and question generation. Then, the unlabeled forestry corpus containing only context is input into the fine-tuned question-answer generation model to generate context-based question and answer pairs, so as to construct a synthetic context, answer, question triplet dataset for fine-tuning the ForestryBERT model.

4. The method for constructing a forestry pre-training language model based on continuous learning according to claim 1, characterized in that: Step 3.2, fine-tuning, includes step 3.2.1, text classification task: Text classification is an automatic process of classifying text data according to predefined categories. It focuses on single-label multi-category classification tasks and is divided into two subtasks according to different classification objectives: samples of the same topic are regarded as different categories, and each topic is further subdivided; samples of the same topic are regarded as the same category and directly classified according to three topics. The first subtask fine-tunes the three models on text classification datasets of different topics to observe whether the models generated in the pre-training phase can effectively retain old knowledge while learning new knowledge. The second subtask only fine-tunes the ForestryBERT model on a forestry text classification dataset containing three topics to observe its performance in completing forestry tasks.

5. The method for constructing a forestry pre-training language model based on continuous learning according to claim 1, characterized in that: The fine-tuning of step 3.2 includes step 3.2.2, extraction question answering task classification: In order to comprehensively evaluate whether the ForestryBERT model has effectively learned forestry expertise, the model is further fine-tuned using the constructed forestry extraction question answering dataset to complete the extraction question answering task. The model automatically extracts relevant information and provides answers based on the given text and questions.

Images