Knowledge extraction and fusion method and system based on domain text classification

Abstract

The invention discloses a knowledge extraction and knowledge fusion method and system based on domain text classification. The method comprises the following steps: S1, constructing a domain knowledge text data set; s2, constructing a ChatGPT-based domain knowledge extraction framework, and extracting domain knowledge contained in the domain knowledge text data set in the step S1; and S3, constructing a multi-task learning knowledge fusion framework based on comparative learning and word classification, and fusing the domain knowledge obtained in the step S2 with the deep language model. According to the method, the acquisition difficulty of high-quality domain knowledge is reduced, and meanwhile, the fusion degree of the language model and the domain knowledge is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to natural language processing technology. The present invention performs domain knowledge extraction and knowledge fusion on domain texts, and specifically provides a knowledge extraction and knowledge fusion method and system based on domain text classification. Background Art

[0002] Domain text classification is an important task in natural language processing, which aims to classify the specific domain or category to which the text belongs based on its content. Different from general text classification, domain text classification focuses on the unique language patterns and features in specific fields (such as medical, legal, financial, news, etc.). It is a task to assign input text to one or more categories in a predefined set of categories. Its goal is to improve the accuracy of classification by utilizing the language patterns, terminology, semantic relations and other features of a specific domain. However, domain text classification faces two major difficulties: the sparsity of domain data and the particularity of domain-specific terminology and language patterns.

[0003] The sparsity of domain data is due to the scarcity of annotated data in specific fields, especially the high cost of obtaining high-quality labeled data, which means that it is difficult to acquire domain knowledge; the particularity of domain-specific terminology and language patterns is due to the fact that different fields have different professional terminology and text features, and even the same word has different meanings in different fields, which means that the integration of domain knowledge is difficult.

[0004] Existing text classification tasks rely heavily on deep learning representation methods, especially large language models such as BERT, RoBERTa, and T5. Even though these models can capture richer text features and context features than traditional methods, simple downstream fine-tuning is not enough to capture deeper domain text features. At the same time, due to the sparsity of domain knowledge, it is difficult for deep language models to integrate more domain knowledge. Summary of the invention

[0005] In view of the above problems existing in the prior art, the present invention proposes a method and system for domain knowledge extraction and knowledge fusion based on domain text classification, which involves the reasonable construction of a large language model interaction paradigm, as well as contrastive learning based on word dimensions and multi-task learning of word classification. The purpose of the present invention is to reduce the difficulty of acquiring high-quality domain knowledge and improve the degree of integration of language models and domain knowledge.

[0006] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] The knowledge extraction and knowledge fusion method based on domain text classification is as follows:

[0008] S1. Build a domain knowledge text dataset to promote the extraction process of domain knowledge and the training and fusion process of domain knowledge;

[0009] S2. Build a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset in step S1;

[0010] S3. Construct a multi-task learning knowledge fusion framework based on contrastive learning and word classification to fuse the domain knowledge obtained in step S2 with the deep language model.

[0011] Preferably, the quality of the training corpus determines the training effect of the model to a large extent. In the present invention, in order to improve the quality of the domain text dataset, the summary text of Baidu Encyclopedia is preferably used as the data source, and the summary text with higher information density is preferably collected. The text classification model is designed based on the domain category associated words to construct the knowledge text datasets in different fields.

[0012] Preferably, in step S1, the method for constructing the domain knowledge text dataset includes the following steps:

[0013] S1.1, original corpus construction, using Baidu Encyclopedia entries and corresponding abstract texts as the original corpus source;

[0014] S1.2. Classification of original corpus: Classify the summary text into categories according to Baidu Encyclopedia's definition of terms, and reduce the overlap between different terms by merging categories. For example, ancient events and ancient wars can be classified into the history category.

[0015] S1.3, original word list construction, use jieba word segmentation to segment the acquired knowledge text, and count several words with high frequency of occurrence to construct the original word list;

[0016] Further preferably, in step S1.2, common keywords and common categories in the field can be constructed according to the required field, and the common keywords and common categories can be used to extract field-related data text from the original unclassified data. At the same time, the work of category merging can be reduced. For example, in the medical field, common keywords such as treatment, cure, and illness can be constructed, and the frequency of occurrence of these keywords in the text data can be calculated to determine whether the data belongs to the medical field.

[0017] Further preferably, in step S1.3, in order to obtain an original vocabulary of suitable size and filter stop words (prepositions, pronouns, conjunctions, etc.), it is necessary to set a suitable frequency threshold and stop word vocabulary. First, the initial frequency threshold is calculated, and the vocabulary is filtered according to the initial frequency threshold; then a stop word vocabulary is constructed: the vocabulary with a default length of 1 is a stop word, and in addition, the collected Chinese stop word list, Harbin Institute of Technology stop word list, and Baidu stop word list are used to construct a stop word list; finally, the stop word vocabulary is used to filter the original vocabulary, and the size of the filtered vocabulary is selected to select whether it is necessary to reduce the frequency threshold to increase the size of the original vocabulary, if the vocabulary size meets the requirements, the construction is completed, otherwise the frequency threshold is reduced to increase the size of the original vocabulary; wherein, the calculation formula of the initial frequency threshold is as follows:

[0018]

[0019] The calculation formula for the vocabulary size criterion is as follows:

[0020]

[0021] Where: L is the number of characters in the original corpus, N is the number of sentences in the original corpus, aver_l(term) is the average word length of the public vocabulary, ρ is the average weight of the estimated word in the entire sentence, the default value is 0.2, T is the vocabulary set, M is the vocabulary size, F is the frequency of occurrence of the word, len(T i ) is the length of the i-th word in the vocabulary, and θ is the average weight of the expected vocabulary in the entire sentence, which defaults to 0.15.

[0022] The above method of constructing domain knowledge text dataset has the following advantages:

[0023] 1) Since the abstract part of Baidu Encyclopedia is used as the data source, the quality and knowledge density of the original corpus can be guaranteed to a great extent, that is, the knowledge density of the abstract part is higher than that of the main text;

[0024] 2) The iterative vocabulary construction method proposed in the present invention can construct a vocabulary of suitable size, that is, the proportion of words in the vocabulary in the original corpus can be guaranteed;

[0025] 3) The present invention has certain expansibility. The method of obtaining knowledge data from Baidu Encyclopedia can be extended to other data sources, and the vocabulary construction method can be applied to any corpus.

[0026] The present invention also proposes a knowledge extraction framework based on ChatGPT, which mainly includes three steps: first, constructing a suitable instruction paradigm to interact with ChatGPT to further screen the vocabulary obtained in step 1; then constructing an instruction paradigm to interact with ChatGPT to construct the classification category corresponding to the vocabulary, and integrating the obtained category range; finally, using the integrated category range to construct a constrained instruction paradigm, and using the instruction paradigm to interact with ChatGPT to construct the category classification knowledge of the vocabulary for the subsequent knowledge fusion step 3, further improving the model's ability to capture domain features.

[0027] Preferably, in step S2, the knowledge extraction method based on ChatGPT is as follows: Figure 1 As shown, the specific steps include:

[0028] S2.1, based on the vocabulary constructed in step S1, use ChatGPT to further filter the vocabulary and select words that are highly relevant to the field;

[0029] S2.2, based on the domain vocabulary screened out in step S2.1, use ChatGPT to subdivide and classify the vocabulary within the domain;

[0030] S2.3. In order to reduce the overlap between the classification categories obtained in step S2.2, the classification categories need to be merged and integrated. For example, hematology and cardiovascular medicine can be merged into internal medicine.

[0031] S2.4. Use the domain classification categories obtained in step S2.3 as the generation constraints of ChatGPT, and use this constraint to reclassify the vocabulary.

[0032] Further preferably, in step S2.1, an interactive paradigm is used, for example: I now have several words, and you need to classify the words I provide according to their relevance to a certain field. The specific classification categories are highly relevant, medium, and irrelevant. According to the categories fed back by ChatGPT, highly relevant words are screened out to form a new field vocabulary; in step S2.2, an interactive paradigm is used: assuming that you are an expert in a certain field, please classify the words I give you into sub-fields under the field, and the feedback format is [word: category]. This step can obtain a preliminary category classification within the field, but since there is no category constraint on ChatGPT, the preliminary classification categories are relatively scattered and unbalanced, that is, the data under each category is relatively sparse and the data volume is unbalanced.

[0033] In order to solve this problem, it is necessary to integrate categories in step S2.3, integrate the preliminary classification categories generated by ChatGPT into new classification category ranges, such as hematology, cardiovascular medicine, nephrology, etc. can be merged into internal medicine categories. The new classification categories will not directly replace the original classification categories, but the new classification categories will be used as the generation constraints of ChatGPT for step S2.4; in step S2.4, an interactive paradigm is generated based on the constraints generated in the above steps: Assuming that you are an expert in a certain field, I will give you several words and category ranges about this field. You need to classify the provided words according to the category range I provide, and the return format is required to be [vocabulary: category]. Through this step, the final domain knowledge data can be obtained, including the domain vocabulary and the corresponding domain classification.

[0034] Based on the domain knowledge acquired in step S2, the present invention proposes a multi-task learning knowledge fusion framework based on contrastive learning and word classification to fuse domain knowledge, specifically including:

[0035] According to the format of the domain knowledge obtained in step S2, a multi-task learning knowledge fusion framework based on contrastive learning and word classification is designed. The currently commonly used deep language models BERT or RoBERTa can be used as the basic deep language model. The domain knowledge obtained in step S2 is fused with the basic model using the multi-task learning knowledge fusion framework to improve the performance of the model in different domain scenarios. The multi-task learning knowledge fusion framework mainly includes three learning tasks: full-word masking task, word contrast learning task, and word classification task. First, the full-word masking task changes the masking unit of the random masking task from character to word. For example, the full-word masking requires that inflammation be masked as [MASK][MASK] instead of inflammation[MASK]. Global information is used to predict the masked content, aiming to help the model learn the meaning, usage, and contextual relationship of vocabulary, thereby promoting the model to adapt to specific domain contexts.

[0036] Further preferably, in step S3, the specific design method of the word contrast learning task is as follows:

[0037] Based on the full-word mask data, given sentence A{x1,x2,x3,x4,x5,x6,x7}, mask x4 and x5 to get sentence A1{x1,x2,x3,[MASK],x6,x7}, use the model output vector of vocabulary {x4,x5} as the positive sample of the mask output vector corresponding to sentence A1, and use other vocabulary in the same batch data as the negative sample of the mask output vector corresponding to sentence A1; the calculation formula of the contrastive learning loss function is as follows:

[0038]

[0039] Where: t is [t1, t2, t3, ..., tN ], N represents the number of [MASK] in the same training batch, f(t i ) represents the text representation vector of the i-th [MASK] represented by the model; t′ is [t′1, t′2, t′3, …, t′ N ],f(t′ i ) represents the text representation vector of the real label corresponding to the i-th [MASK] represented by the model, that is, f(t i ) is a positive sample vector; τ is a set hyperparameter with a default value of 0.01; sim(h1,h2) represents the calculation of the cosine similarity of the two vectors h1 and h2. The calculation formula is as follows:

[0040]

[0041] This contrastive learning task can help the model gradually learn the semantic similarity and semantic relevance between domain words during the training process, thereby enhancing the model's domain text understanding ability.

[0042] Further preferably, the specific design method of the word classification task in step S3 is as follows:

[0043] Based on the full word mask data, for example, given a sentence A{x1,x2,x3,x4,x5,x6,x7}, x4 and x5 are masked to obtain sentence A1{x1,x2,x3,[MASK],x6,x7}, and the output vector of [MASK] is used as the input of the classification layer. At the same time, the classification category information {x4,x5} obtained in step S2 is used as the classification label to calculate the word classification task loss; the calculation formula of the word classification task loss function is as follows:

[0044]

[0045] In the formula, C represents the number of categories, y represents the true label vector of the current data, and the dimension is 1×C. i Represents the true label value of the i-th dimension of the data, Represents the predicted label vector of the current data, that is, the classification vector of the data after passing through the model and the classification layer. Represents the predicted label value of the i-th dimension of the data. The word classification task can provide additional contextual understanding and domain knowledge during the training process, thereby improving the model's ability to distinguish different categories in a specific field.

[0046] Preferably, the training framework of step S3, i.e., the multi-task learning knowledge fusion framework, is as follows: Figure 2 As shown; the calculation formula of the total loss function of the training framework is as follows:

[0047] L overall =LWWM +L CL +L WC

[0048] First, we obtain the full-word masking loss L from the full-word masking task WWM , which can help the model learn the meaning, usage and relationship of words in context, thereby promoting the model to adapt to the context of a specific field; then obtain the contrastive learning loss L from the word contrastive learning task CL , this task can gradually help the model learn the semantic similarity and contextual relevance between words during the training process, thereby improving its ability to understand text; finally, the classification loss L is obtained from the word classification task WC , this loss function provides additional contextual understanding and domain knowledge, thereby improving the model's ability to distinguish different categories in a specific domain.

[0049] The present invention also discloses a knowledge extraction and knowledge fusion system based on domain text classification, which is used to execute the above method and includes the following modules:

[0050] Dataset construction module: construct domain knowledge text dataset;

[0051] Extraction module: Build a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset;

[0052] Fusion module: Build a multi-task learning knowledge fusion framework based on contrastive learning and word classification to integrate the acquired domain knowledge with the deep language model.

[0053] Compared with the prior art, the present invention has many advantages, including the following two key points:

[0054] 1) Reduce the difficulty of acquiring high-quality domain knowledge: The domain knowledge extraction method based on ChatGPT proposed in the present invention can extract high-quality domain knowledge from the original corpus, which can be applicable to the original corpus in multiple fields;

[0055] 2) Enrich the model’s domain knowledge information: Use a multi-task learning framework to integrate high-quality domain knowledge and information from the domain text itself, and comprehensively utilize multi-dimensional domain knowledge to improve the model’s domain understanding capabilities.

[0056] In summary, the present invention proposes a knowledge domain fusion framework, which includes two parts: a knowledge extraction framework and a knowledge fusion framework. Among them, the knowledge extraction framework is based on the data generation capability of the current large language model, such as using ChatGPT to extract the domain knowledge contained in the original text. Specifically, the interactive constraints are first generated through ChatGPT, and then the constraints are used to build high-quality domain knowledge; the knowledge fusion framework is based on the multi-task learning framework, and designs a joint training framework of three tasks: full-word masking task, word comparison learning task and word classification task. It makes great use of the domain words containing high-density domain knowledge, and improves the model's domain understanding and reasoning ability. The present invention constructs a knowledge extraction method and a knowledge fusion method based on the word dimension, aiming to overcome the sparsity of domain knowledge and improve the model's domain knowledge understanding ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of the domain knowledge extraction method based on ChatGPT proposed in the preferred embodiment of the present invention.

[0058] Figure 2 It is a framework diagram of domain knowledge fusion training based on multi-task learning proposed in the preferred embodiment of the present invention.

[0059] Figure 3 It is a block diagram of a knowledge extraction and knowledge fusion system based on domain text classification according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following is an explanation of the implementation method of the present invention by specific embodiments. Those skilled in the art can easily understand the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied by other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the features in the following embodiments can be combined with each other without conflict.

[0061] Aiming at the problem of domain text classification in a specific domain, the present invention extracts domain knowledge from the original corpus based on ChatGPT, designs a multi-task learning framework based on word dimension, integrates domain knowledge into the language model, and improves the model's domain text understanding and reasoning capabilities.

[0062] like Figure 1-2 As shown, this embodiment provides a domain knowledge extraction method based on ChatGPT and a domain knowledge fusion method based on multi-task learning, which is carried out in the following steps:

[0063] S1. Build a domain knowledge text dataset to promote the extraction process of domain knowledge and the training and fusion process of domain knowledge;

[0064] S2. Build a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset in step S1;

[0065] S3. Construct a multi-task learning knowledge fusion framework based on contrastive learning and word classification to fuse the domain knowledge obtained in step S2 with the deep language model.

[0066] Each step is described in more detail below.

[0067] In step S1, a knowledge text data set is constructed according to the application field. Since the quality of Baidu Encyclopedia data is relatively high, this embodiment uses Baidu Encyclopedia data as the data source of knowledge text data. At the same time, because the abstract part in Baidu Encyclopedia data has a higher knowledge density than the main text part, this embodiment uses the abstract text in Baidu Encyclopedia to construct knowledge text data; after the original corpus is constructed, it depends on the classification category of the abstract terms in Baidu Encyclopedia or constructs the field keywords by itself to extract the original corpus data related to the field. For example, for the medical field, it can be determined whether the abstract text is related to medical care based on whether the category of the abstract terms in the existing Baidu Encyclopedia is related to medical care or a medical common word list is constructed. Through this step, the original corpus data related to the application field can be obtained; based on the original corpus of the field, the original corpus is segmented using the jieba tool to obtain the vocabulary and the corresponding frequency of occurrence of the vocabulary, and the selection is based on the frequency of occurrence. The specific steps are as follows:

[0068] 1) Calculate the initial frequency threshold, and filter the domain words with a frequency lower than the threshold according to the frequency threshold. The calculation method of the initial frequency threshold is as follows:

[0069]

[0070] Where L is the number of characters in the original corpus, N is the number of sentences in the original corpus, aver_l(term) is the average word length of the public vocabulary, and ρ is the average weight of the estimated word in the entire sentence, which defaults to 0.2.

[0071] 2) Construct a stop word vocabulary, which mainly includes prepositions, pronouns, conjunctions, etc. that do not contain domain information. First, the default length of 1 is a stop word. Then, the stop word vocabulary used in this example is constructed based on the three Chinese stop word lists: the Chinese stop word list, the Harbin Institute of Technology stop word list, and the Baidu stop word list. The constructed stop word vocabulary is used to further filter the domain vocabulary to improve the domain knowledge density of the vocabulary;

[0072] 3) Evaluate the size of the domain vocabulary. The specific evaluation method is as follows:

[0073]

[0074] Where: L is the number of characters in the original corpus, T is the vocabulary set, M is the vocabulary size, F is the frequency of occurrence of the word, len(T i ) is the length of the i-th word in the vocabulary, and θ is the average weight of the expected vocabulary in the entire sentence, which defaults to 0.15. When the calculated value is greater than 1, it means that the constructed domain vocabulary accounts for a higher proportion of the entire corpus than θ, that is, the vocabulary size is appropriate. If it is less than 1, then it is necessary to lower the frequency threshold and continue to filter stop words. This process needs to continue until V(L,T,M,F)>1 is satisfied.

[0075] In step S2, refer to Figure 1 , this embodiment designs a method for constructing domain knowledge based on ChatGPT, which is used to improve the quality and quantity of domain knowledge and promote the training process of subsequent models. First, it is necessary to screen out domain vocabulary with high domain knowledge density from the vocabulary obtained in step S1. The main method is to construct a reasonable interaction paradigm to interact with ChatGPT, such as the paradigm used in this example: I now have several words, and you need to classify the several words I provide according to the degree of relevance to a certain field. The specific classification categories are highly relevant, medium, and irrelevant. Through this interactive paradigm, ChatGPT can obtain the degree of domain relevance of the provided vocabulary, and screen out highly relevant vocabulary to form a new domain vocabulary with high-density domain knowledge. The domain vocabulary is a preliminary form of domain knowledge obtained in this example, which can be used as a mask unit in the full-word masking task, that is, using global context information to predict the vocabulary in this field, which can help the model learn the meaning, usage, and relationship in context of vocabulary, thereby promoting the model to adapt to the context of a specific field.

[0076] Although the domain vocabulary obtained in the previous step contains rich domain features, it lacks the distinguishing information between domain vocabulary. In order to obtain domain knowledge with high discrimination within the domain, this embodiment adopts the paradigm "assuming that you are an expert in a certain field, please classify the several words I give you into sub-domains under the field, and the feedback format is [word: category]" to interact with ChatGPT to obtain preliminary classification information of domain vocabulary; through this step, this example can obtain classification knowledge with high discrimination, but because there is no constraint on ChatGPT except the field in the interactive paradigm, the obtained classification knowledge is relatively scattered, that is, the data under each category will be relatively sparse, and the overlap between each category is large, so it is necessary to manually integrate the obtained categories within the field, and integrate several categories into new classification categories to reduce the sparsity of data under the category and the overlap between categories, that is, through this step, a new, low-overlapping, and wide-coverage classification category can be obtained; this example then uses the classification category to construct an interactive paradigm with constraints: assuming that you are an expert in a certain field, I will give you several words and category ranges about the field, you need to classify the provided words according to the category range I provide, and the return format is required to be [word: category]. The final vocabulary classification knowledge is obtained by interacting with ChatGPT through this paradigm.

[0077] In step S3, refer to Figure 2 This embodiment designs a knowledge fusion method based on multi-task learning, which mainly includes three training tasks: full-word masking task, word contrast learning task and word classification task. The full-word masking task uses the domain vocabulary obtained in step S2, and modifies the masking unit to domain vocabulary on the basis of the random masking task, that is, if there are words included in the domain vocabulary in the current sentence, then the vocabulary is masked at a masking rate of 15%. If the masking rate is less than 15% after masking the vocabulary, then the random words are masked according to random masking until the masking rate reaches 15%, and the context information in the sentence is used to predict the masked vocabulary or words, thereby promoting the model to adapt to the context of a specific domain.

[0078] The word contrast learning task uses the contrast learning task to constrain the text representation learned by the model on the basis of the full-word masking task. That is, the currently masked word can be predicted as multiple words. In order to constrain the range of predicted words and improve the robustness of the model, this example uses the true label as the positive sample at the mask mark, and uses other masked words in the same batch of samples as the negative sample of the current masked word. The loss function of this task is calculated as follows:

[0079]

[0080] Where: t is [t1, t2, t3, ..., t N], N represents the number of [MASK] in the same training batch, f(t i ) represents the text representation vector of the i-th [MASK] represented by the model; t′ is [t′1, t′2, t′3, …, t′ N ],f(t′ i ) represents the text representation vector of the real label corresponding to the i-th [MASK] represented by the model, that is, f(t i ) is a positive sample vector; τ is a set hyperparameter with a default value of 0.01; sim(h1,h2) represents the calculation of the cosine similarity of the two vectors h1 and h2. The calculation formula is as follows:

[0081]

[0082] This contrastive learning task can not only constrain the training process of the whole-word masking task, but also help the model gradually learn the semantic similarity and semantic relevance between domain words during the training process, thereby enhancing the model's domain text understanding ability.

[0083] Although the full-word masking task and contrastive learning task utilize more domain information than traditional tasks, they only utilize information of major domain categories, that is, they only use the application domain to constrain the model training, but do not use the domain information within the domain. Therefore, in the end, this example uses the internal classification information of the domain obtained in step S2 to constrain the model training. That is, on the basis of the above two tasks, the classification category information of the masked words in the domain is provided during training. This information is used to further constrain the training process of the model and expand the distinction between domain words. The specific calculation method of the loss function of the classification task is as follows:

[0084]

[0085] In the formula, C represents the number of categories, y represents the true label vector of the current data, and the dimension is 1×C. i Represents the true label value of the i-th dimension of the data, Represents the predicted label vector of the current data, that is, the classification vector of the data after passing through the model and the classification layer. Represents the predicted label value of the i-th dimension of the data. This loss function can further constrain the model, improve the model's feature distinction ability in this field, and thus promote the model's performance in this field.

[0086] Through the above three training tasks, the domain knowledge extracted in step S2 can be well integrated into the model, improving the model's domain knowledge understanding and reasoning capabilities. At the same time, the domain knowledge is used to constrain the model's training process to improve the robustness of the model.

[0087] In step S3, the total loss function of the training framework is calculated as follows:

[0088] L overall =L WWM +L CL +L WC

[0089] First, obtain the full-word mask loss L from the full-word mask task WWM , which can help the model learn the meaning, usage and relationship of words in context, thereby promoting the model to adapt to the context of a specific field; then obtain the contrastive learning loss L from the word contrastive learning task CL , this task can gradually help the model learn the semantic similarity and contextual relevance between words during the training process, thereby improving its ability to understand text; finally, the classification loss L is obtained from the word classification task WC , this loss function provides additional contextual understanding and domain knowledge, thereby improving the model's ability to distinguish different categories in a specific domain.

[0090] Table 1 is a comparison table of classification task results of the present invention on test data in multiple fields.

[0091] Table 1

[0092]

[0093] Table 2 is a comparison table of retrieval task results of the present invention on test data in multiple fields.

[0094] Table 2

[0095]

[0096] Table 1 above is an evaluation of the framework proposed in the present invention on text classification tasks in four fields compared with other existing methods, and the evaluation index is the F1 score; Table 2 is an evaluation of the framework proposed in the present invention on text retrieval tasks in four fields compared with other existing methods, and the evaluation index is the MRR score. From these two tables, it can be seen that the knowledge fusion framework proposed in the present invention is better than other existing methods in multiple tasks in multiple fields, and it also proves the effectiveness of the knowledge obtained by the knowledge extraction framework of the present invention, because compared with the benchmark model, the model trained by the present invention using knowledge is better.

[0097] In summary, the preferred embodiment of the present invention aims at the problem of domain text classification in a specific domain, and proposes a domain knowledge extraction method based on ChatGPT and a domain knowledge fusion method based on multi-task learning. It uses a reasonable paradigm construction process to constrain ChatGPT to generate high-quality, fixed-format domain knowledge, and uses multiple training tasks to perform mutual training constraints to improve the model's domain knowledge understanding and reasoning capabilities, while improving the robustness of the model and enhancing the model's performance in domain classification tasks.

[0098] like Figure 3 As shown, this embodiment discloses a knowledge extraction and knowledge fusion system based on domain text classification, which is used to execute the above method embodiment, and includes the following modules:

[0099] Dataset construction module: construct domain knowledge text dataset;

[0100] Extraction module: Build a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset;

[0101] Fusion module: Build a multi-task learning knowledge fusion framework based on contrastive learning and word classification to integrate the acquired domain knowledge with the deep language model.

[0102] For other contents of this embodiment, please refer to the above method embodiment.

[0103] The above embodiments are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art of the present invention may make other supplements or modifications to the specific embodiments described, or replace them with other methods. Therefore, although the above embodiments make the purpose and advantages of the present invention clearer, the present invention is not limited to the above embodiments. Therefore, this specification should not be understood as limiting the present invention.

Claims

1. Knowledge extraction and knowledge fusion method based on domain text classification, characterized by Follow these steps: S1. Build domain knowledge text dataset; S2. Build a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset in step S1; S3. Construct a multi-task learning knowledge fusion framework based on contrastive learning and word classification to fuse the domain knowledge obtained in step S2 with the deep language model.

2. The method for knowledge extraction and knowledge fusion based on domain text classification as claimed in claim 1, characterized in that: In step S1, the construction of the domain knowledge text dataset includes the following steps: S1.1, original corpus construction, using entries and corresponding abstract texts as original corpus sources; S1.2, original corpus classification, divide the summary text into categories according to the category definition of the entry, and merge the categories; S1.3, construct the original vocabulary, segment the acquired knowledge text, and count the words that appear above the set frequency to construct the original vocabulary.

3. The method for knowledge extraction and knowledge fusion based on domain text classification as claimed in claim 2, characterized in that In step S1.2, common keywords and common categories are constructed according to the required domain, and domain-related data text is extracted from the original unclassified data using the common keywords and common categories.

4. The method for knowledge extraction and knowledge fusion based on domain text classification as claimed in claim 2, characterized in that In step S1.3, in order to obtain the original vocabulary and filter out stop words at the same time, it is necessary to set the frequency threshold and the stop word vocabulary; first, calculate the initial frequency threshold, and filter the vocabulary according to the initial frequency threshold; Then construct a stop word list, including the default words with a length of 1 as stop words and the stop word list collected and constructed; Finally, the stop word vocabulary is used to filter the original vocabulary. According to the size of the filtered vocabulary, it is selected whether the frequency threshold needs to be lowered to increase the size of the original vocabulary. If the vocabulary size meets the requirements, the construction is completed. Otherwise, the frequency threshold is lowered to increase the size of the original vocabulary. The calculation formula of the initial frequency threshold is as follows: The calculation formula for the vocabulary size criterion is as follows: Where: L is the number of characters in the original corpus, N is the number of sentences in the original corpus, aver_l(term) is the average word length of the public vocabulary, ρ is the average weight of the estimated word in the entire sentence, T is the vocabulary set, M is the vocabulary size, F is the frequency of occurrence of the word, len(T i ) is the length of the i-th word in the vocabulary, and θ is the average weight of the expected vocabulary in the entire sentence.

5. The method for knowledge extraction and knowledge fusion based on domain text classification as claimed in claim 1, characterized in that: In step S2, knowledge extraction is specifically as follows: S2.1, based on the vocabulary constructed in step S1, use ChatGPT to filter the vocabulary and select words that are highly relevant to the field; S2.2, based on the domain vocabulary screened in step S2.1, use ChatGPT to subdivide and classify the vocabulary within the domain; S2.3, merge and integrate the classification categories; S2.

4. Use the domain classification categories obtained in step S2.3 to constrain ChatGPT, and use the constraints to reclassify the vocabulary.

6. The method for knowledge extraction and knowledge fusion based on domain text classification according to any one of claims 1 to 5, characterized in that: In step S3, according to the format of the domain knowledge obtained in step S2, a multi-task learning knowledge fusion framework based on contrastive learning and word classification is constructed. The framework mainly includes three learning tasks: a full-word masking task, a word contrastive learning task, and a word classification task; the full-word masking task changes the masking unit of the random masking task from characters to words.

7. The method for knowledge extraction and knowledge fusion based on domain text classification as claimed in claim 6, characterized in that: In step S3, the word contrast learning task is as follows: Based on the full-word masking task, given a sentence A{x1,x2,x3,x4,x5,x6,x7}, x4 and x5 are masked to obtain sentence A1{x1,x2,x3,[MASK],x6,x7}, the model output vector of the vocabulary {x4,x5} is used as the positive sample of the masked output vector corresponding to the A1 sentence, and other words in the same batch of data are used as negative samples of the masked output vector corresponding to the A1 sentence; the calculation formula of the contrastive learning loss function is as follows: Where: t is [t1, t2, t3, ..., t N ], N represents the number of [MASK] in the same training batch, f(t i ) represents the text representation vector of the i-th [MASK] represented by the model; t′ is [t′1, t′2, t′3, …, t′ N ],f(t′ i ) represents the text representation vector of the real label corresponding to the i-th [MASK] represented by the model, that is, f(t i ) is a positive sample vector; τ is a set hyperparameter; sim(h1,h2) represents the calculation of the cosine similarity of the two vectors h1 and h2, and the calculation formula is as follows:

8. The method for knowledge extraction and knowledge fusion based on domain text classification as claimed in claim 6, characterized in that: In step S3, the word classification task is as follows: Based on the full-word masking task, given a sentence A{x1,x2,x3,x4,x5,x6,x7}, x4 and x5 are masked to obtain sentence A1{x1,x2,x3,[MASK],x6,x7}, and the output vector of [MASK] is used as the input of the classification layer. At the same time, the classification category information of {x4,x5} obtained in step S2 is used as the classification label to calculate the word classification task loss; the calculation formula of the word classification task loss function is as follows: In the formula, C represents the number of categories, y represents the true label vector of the current data, and the dimension is 1×C. i Represents the true label value of the i-th dimension of the data, Represents the predicted label vector of the current data, that is, the classification vector of the data after passing through the model and the classification layer. Represents the predicted label value of the i-th dimension of the data.

9. The method for knowledge extraction and knowledge fusion based on domain text classification as claimed in claim 6, characterized in that: In step S3, the calculation formula of the total loss function of the multi-task learning knowledge fusion framework is as follows: L overall =L WWM +L CL +L WC First, obtain the full-word masking loss L from the full-word masking task WWM , and then obtain the contrastive learning loss L from the word contrastive learning task CL , and finally obtain the classification loss L from the word classification task WC .

10. A knowledge extraction and knowledge fusion system based on domain text classification, used to execute the method according to any one of claims 1 to 9, characterized in that Includes the following modules: Dataset construction module: construct domain knowledge text dataset; Extraction module: Build a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset; Fusion module: Build a multi-task learning knowledge fusion framework based on contrastive learning and word classification to integrate the acquired domain knowledge with the deep language model.

Images