Knowledge extraction and fusion method and system based on domain text classification

Abstract

The application discloses a knowledge extraction and knowledge fusion method and system based on field text classification, and the method comprises the following steps: S1, constructing a field knowledge text dataset; S2, constructing a field knowledge extraction framework based on ChatGPT, and extracting the field knowledge contained in the field knowledge text dataset in step S1; S3, constructing a multi-task learning knowledge fusion framework based on contrast learning and word classification, and fusing the field knowledge obtained in step S2 with a deep language model. The application reduces the difficulty of obtaining high-quality field knowledge, and improves the fusion degree of the language model and the field knowledge.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to natural language processing technology. It extracts and fuses domain knowledge from domain texts, specifically a method and system for knowledge extraction and fusion based on domain text classification. Background Technology

[0002] Domain-specific text classification is an important task in natural language processing, aiming to categorize text based on its content within a specific domain or category. Unlike general text classification, domain-specific text classification focuses on the unique language patterns and features of a specific domain (such as medical, legal, financial, or news). It involves assigning input text to one or more categories from a predefined set of categories, with the goal of improving classification accuracy by leveraging domain-specific language patterns, terminology, and semantic relationships. However, domain-specific text classification faces two main challenges: the sparsity of domain data and the specificity of domain-specific terminology and language patterns.

[0003] The sparsity of domain data is due to the scarcity of labeled data in specific domains, especially the high cost of acquiring high-quality labeled data, which means that acquiring domain knowledge is quite difficult; the particularity of domain-specific terms and language patterns is due to the different professional terms and text features in different domains, and even the same word may have different meanings in different domains, which means that the integration of domain knowledge is quite difficult.

[0004] Existing text classification tasks largely rely on deep learning representation methods, especially large language models such as BERT, RoBERTa, and T5. Even though these models can capture richer text features and contextual features compared to traditional methods, simple downstream fine-tuning is insufficient to capture deeper domain text features. At the same time, due to the sparsity of domain knowledge, it is difficult for deep language models to integrate more domain knowledge. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention proposes a method and system for domain knowledge extraction and fusion based on domain text classification. This involves the rational construction of a language large-scale model interaction paradigm, as well as multi-task learning based on word dimension contrastive learning and word classification. The purpose of this invention is to reduce the difficulty of acquiring high-quality domain knowledge while improving the degree of integration between language models and domain knowledge.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The knowledge extraction and fusion method based on domain text classification follows these steps:

[0008] S1. Construct a domain knowledge text dataset to facilitate the extraction and training fusion of domain knowledge.

[0009] S2. Construct a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset in step S1;

[0010] S3. Construct a multi-task learning knowledge fusion framework based on contrastive learning and word classification, and fuse the domain knowledge obtained in step S2 with the deep language model.

[0011] Preferably, the quality of the training corpus largely determines the training effect of the model. In this invention, in order to improve the quality of the domain text dataset, it is preferable to use the summary text of Baidu Encyclopedia as the data source, and preferably to collect summary texts with higher information density. Text classification patterns are designed based on domain category-related words to construct knowledge text datasets for different domains.

[0012] Preferably, in step S1, the construction of the domain knowledge text dataset includes the following steps:

[0013] S1.1 Construction of the original corpus: Baidu Encyclopedia entries and corresponding abstract texts were used as the source of the original corpus.

[0014] S1.2. Original corpus classification: Based on the category definitions of Baidu Encyclopedia for the entries, the abstract text is classified into categories. Category merging is used to reduce the overlap between different entries. For example, ancient events and ancient wars can be classified into the history category.

[0015] S1.3 Construction of the original vocabulary: The obtained knowledge text is segmented using jieba word segmentation, and a number of words with high frequency of occurrence are counted to construct the original vocabulary.

[0016] More preferably, in step S1.2, common keywords and common categories can be constructed according to the required field. Common keywords and common categories are used to extract field-related data text from the original unclassified data. At the same time, the work of merging categories can be reduced. For example, in the medical field, common keywords such as treatment, cure, and illness can be constructed, and the frequency of these keywords in the text data can be calculated to determine whether the data belongs to the medical field.

[0017] More preferably, in step S1.3, in order to obtain an original vocabulary of suitable size while filtering stop words (prepositions, pronouns, conjunctions, etc.), it is necessary to set a suitable occurrence frequency threshold and a stop word vocabulary. First, the initial occurrence frequency threshold is calculated, and the vocabulary is filtered based on the initial occurrence frequency threshold; then, a stop word vocabulary is constructed: words with a default length of 1 are considered stop words. In addition, a stop word vocabulary is constructed using collected Chinese stop word lists, Harbin Institute of Technology stop word lists, and Baidu stop word lists; finally, the original vocabulary is filtered using the stop word vocabulary. Based on the size of the filtered vocabulary, it is determined whether to lower the occurrence frequency threshold to increase the size of the original vocabulary. If the vocabulary size meets the requirements, the construction is complete; otherwise, the occurrence frequency threshold is lowered to increase the size of the original vocabulary. The formula for calculating the initial occurrence frequency threshold is as follows:

[0018]

[0019] The formula for calculating the criterion for vocabulary size is as follows:

[0020]

[0021] In the formula: L is the number of characters in the original corpus, N is the number of sentences in the original corpus, aver_l(term) is the average vocabulary length of the public vocabulary, ρ is the estimated average weight of the vocabulary in the whole sentence (default is 0.2), T is the vocabulary set, M is the vocabulary size, F is the frequency of word occurrence, and len(T) is the average length of the vocabulary. i ) represents the length of the i-th word in the vocabulary, and θ represents the expected average weight of the vocabulary in the entire sentence, which defaults to 0.15.

[0022] The above-mentioned method for constructing domain knowledge text datasets has the following advantages:

[0023] 1) By using the abstract of Baidu Encyclopedia as the data source, the quality and knowledge density of the original corpus can be guaranteed to a great extent, that is, the knowledge density of the abstract is higher than that of the main text.

[0024] 2) The iterative vocabulary construction method proposed in this invention can construct a vocabulary of a suitable size, that is, the proportion of words in the vocabulary in the original corpus can be guaranteed;

[0025] 3) This invention has a certain degree of scalability. The method of obtaining knowledge data from Baidu Encyclopedia can be extended to other data sources, and the vocabulary construction method can be applied to any corpus.

[0026] This invention also proposes a knowledge extraction framework based on ChatGPT, which mainly includes three steps: First, a suitable instruction paradigm is constructed to interact with ChatGPT to further filter the vocabulary obtained in step 1; then, an instruction paradigm is constructed to interact with ChatGPT to build the classification categories corresponding to the vocabulary, and the obtained category ranges are integrated; finally, a constrained instruction paradigm is constructed using the integrated category ranges, and the category classification knowledge of the vocabulary is constructed by interacting with ChatGPT using this instruction paradigm for subsequent knowledge fusion step 3, further improving the model's ability to capture domain features.

[0027] Preferably, in step S2, the knowledge extraction method based on ChatGPT follows these steps: Figure 1 As shown, the specific steps include:

[0028] S2.1 Based on the vocabulary list constructed in step S1, ChatGPT is used to further filter the vocabulary list to select words that are highly relevant to the domain.

[0029] S2.2. Based on the domain terminology list selected in step S2.1, ChatGPT is used to further subdivide and classify the terminology list within the domain.

[0030] S2.3 In order to reduce the overlap between the classification categories obtained in step S2.2, it is necessary to merge and integrate the classification categories. For example, hematology and cardiology can be merged into internal medicine.

[0031] S2.4. Use the domain classification categories obtained in step S2.3 as the generation constraints of ChatGPT, and use these constraints to reclassify the vocabulary.

[0032] Further optimized, in step S2.1, an interactive paradigm is used, for example: I have several words, and you need to classify these words according to their relevance to a certain field, specifically into three categories: highly relevant, moderately relevant, and irrelevant. Based on the categories provided by ChatGPT, highly relevant words are selected to form a new domain-specific vocabulary. In step S2.2, an interactive paradigm is used: Assuming you are an expert in a certain field, please classify the words I provide into sub-fields within that field, with the feedback format being [word: category]. This step provides a preliminary category classification within the domain; however, because there are no category constraints on ChatGPT, the preliminary classification is relatively scattered and imbalanced, meaning that the data under each category is sparse and the data volume is unbalanced.

[0033] To solve this problem, it is necessary to perform category integration in step S2.3, integrating the preliminary classification categories generated by ChatGPT into a new classification category range. For example, hematology, cardiology, nephrology, etc. can be merged into the internal medicine category. The new classification category will not directly replace the original classification category, but will be used as a generation constraint for ChatGPT in step S2.4; in step S2.4, an interaction paradigm is generated based on the constraints generated in the above steps: Assume you are an expert in a certain field, and I will provide you with several vocabulary words and category ranges in this field. You need to classify the provided vocabulary words according to the category ranges I provided, and the required return format is [vocabulary: category]. Through this step, the final domain knowledge data can be obtained, including the domain vocabulary and the corresponding domain classification.

[0034] Based on the domain knowledge obtained in step S2, the present invention proposes a multi-task learning knowledge fusion framework based on contrastive learning and word classification to fuse domain knowledge, which specifically includes:

[0035] According to the format of the domain knowledge obtained in step S2, a multi-task learning knowledge fusion framework based on contrastive learning and word classification is designed. The commonly used deep language models BERT or RoBERTa can be used as the basic deep language model, and the domain knowledge obtained in step S2 is fused with the basic model using this multi-task learning knowledge fusion framework to improve the performance of the model in different domain scenarios. This multi-task learning knowledge fusion framework mainly includes three learning tasks: the whole-word masking task, the word contrastive learning task, and the word classification task. First, the whole-word masking task changes the masking unit of the random masking task from a character to a word. For example, the whole-word masking needs to mask "inflammation" as [MASK][MASK] instead of "in[MASK]", and uses global information to predict the masked content, aiming to help the model learn the meaning, usage, and context relationship of vocabulary, so as to promote the model to adapt to a specific domain context.

[0036] Further preferably, in step S3, the specific design method of the word contrastive learning task is as follows:

[0037] Based on the whole-word masking data, a sentence A{x1,x2,x3,x4,x5,x6,x7} is given, and x4 and x5 are masked to obtain sentence A1{x1,x2,x3,[MASK],x6,x7}. The model output vectors of the vocabulary {x4,x5} are used as the positive samples of the corresponding masked output vectors of sentence A1, and the other vocabulary words in the same batch of data are used as the negative samples of the corresponding masked output vectors of sentence A1; among them, the calculation formula of the contrastive learning loss function is as follows:

[0038]

[0039] In the formula: t is [t1,t2,t3,……,tN ], N represents the number of [MASK] in the same training batch, f(t) i ) represents the text representation vector of the i-th [MASK] represented by the model; t′ is [t′1, t′2, t′3, ..., t′] N ],f(t′ i f(t) represents the text representation vector of the true label corresponding to the i-th [MASK] through the model, i.e., f(t) i The positive sample vectors are represented by h1; τ is a hyperparameter that is set, with a default value of 0.01; sim(h1,h2) calculates the cosine similarity between the two vectors h1 and h2, and the calculation formula is as follows:

[0040]

[0041] This contrastive learning task can help the model gradually learn the semantic similarity and semantic relevance between domain words during the training process, thereby enhancing the model's domain text understanding ability.

[0042] A further preferred design for the word classification task in step S3 is as follows:

[0043] Based on the full-word masked data, given the sentence A{x1,x2,x3,x4,x5,x6,x7}, masking x4 and x5 yields the sentence A1{x1,x2,x3,[MASK],x6,x7}. The output vector of [MASK] is used as the input to the classification layer, and the classification information of {x4,x5} obtained in step S2 is used as the classification label to calculate the word classification task loss. The formula for calculating the word classification task loss function is as follows:

[0044]

[0045] In the formula, C represents the number of categories, and y represents the true label vector of the current data, with a dimension of 1×C. i This represents the true label value of the i-th dimension of the data. This represents the predicted label vector of the current data, that is, the classification vector of the data after passing through the model and classification layer. This represents the predicted label value for the i-th dimension of the data. Word classification tasks can provide additional contextual understanding and domain knowledge during training, thereby improving the model's ability to distinguish different categories within a specific domain.

[0046] Preferably, the training framework for specific step S3 is a multi-task learning knowledge fusion framework, such as... Figure 2 As shown; the formula for calculating the total loss function of the training framework is as follows:

[0047] L overall =LWWM +L CL +L WC

[0048] First, obtain the full-word masking loss L from the full-word masking task. WWM This task helps the model learn the meaning, usage, and relationships of words in context, thereby enabling the model to adapt to specific domain contexts; subsequently, the contrastive learning loss L is obtained from the word contrastive learning task. CL This task can gradually help the model learn the semantic similarity and contextual relevance between words during training, thereby improving its ability to understand text; finally, the classification loss L is obtained from the word classification task. WC This loss function provides additional contextual understanding and domain knowledge, thereby improving the model's ability to distinguish different categories in a specific domain.

[0049] This invention also discloses a knowledge extraction and fusion system based on domain text classification for performing the above methods, which includes the following modules:

[0050] Dataset building module: Constructs domain knowledge text datasets;

[0051] Extraction module: Constructs a domain knowledge extraction framework based on ChatGPT to extract domain knowledge contained in domain knowledge text datasets;

[0052] Fusion Module: Constructs a multi-task learning knowledge fusion framework based on contrastive learning and word classification, which integrates the acquired domain knowledge with the deep language model.

[0053] Compared with the prior art, the present invention has several advantages, specifically including the following two key points:

[0054] 1) Reduce the difficulty of acquiring high-quality domain knowledge: This invention proposes a domain knowledge extraction method based on ChatGPT, which can extract high-quality domain knowledge from the original corpus and can be applied to the original corpus of multiple domains;

[0055] 2) Enrich the model's domain knowledge information: Integrate high-quality domain knowledge and information from the domain text itself using a multi-task learning framework, and comprehensively utilize multi-dimensional domain knowledge to improve the model's domain understanding ability.

[0056] In summary, this invention proposes a knowledge domain fusion framework, comprising two parts: a knowledge extraction framework and a knowledge fusion framework. The knowledge extraction framework leverages the data generation capabilities of current large-scale language models, such as using ChatGPT to extract domain knowledge embedded in the original text. Specifically, it first generates interaction constraints using ChatGPT and then uses these constraints to construct high-quality domain knowledge. The knowledge fusion framework is based on a multi-task learning framework, designing a joint training framework for three tasks: full-word masking, word comparison learning, and word classification. This maximizes the utilization of domain words containing high-density domain knowledge, improving the model's domain understanding and reasoning capabilities. This invention constructs knowledge extraction and fusion methods based on word dimensions, aiming to overcome the sparsity of domain knowledge and improve the model's domain knowledge understanding ability. Attached Figure Description

[0057] Figure 1 This is a flowchart of the domain knowledge extraction method based on ChatGPT proposed in a preferred embodiment of the present invention.

[0058] Figure 2 This is a diagram of a domain knowledge fusion training framework based on multi-task learning proposed in a preferred embodiment of the present invention.

[0059] Figure 3 This is a block diagram of a knowledge extraction and knowledge fusion system based on domain text classification, according to a preferred embodiment of the present invention. Detailed Implementation

[0060] The following specific embodiments illustrate the implementation method of the present invention. Those skilled in the art can easily understand the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific methods, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the features in the following embodiments can be combined with each other.

[0061] This invention addresses the domain-specific text classification problem by extracting domain knowledge from the original corpus using ChatGPT and designing a word-dimensional multi-task learning framework to integrate domain knowledge into the language model, thereby improving the model's domain text understanding and reasoning capabilities.

[0062] like Figure 1-2 As shown, this embodiment provides a domain knowledge extraction method based on ChatGPT and a domain knowledge fusion method based on multi-task learning, which proceeds as follows:

[0063] S1. Construct a domain knowledge text dataset to facilitate the extraction and training fusion of domain knowledge.

[0064] S2. Construct a domain knowledge extraction framework based on ChatGPT to extract the domain knowledge contained in the domain knowledge text dataset in step S1;

[0065] S3. Construct a multi-task learning knowledge fusion framework based on contrastive learning and word classification, and fuse the domain knowledge obtained in step S2 with the deep language model.

[0066] The following provides a more detailed explanation of each step.

[0067] In step S1, a knowledge text dataset is constructed based on the application domain. Since Baidu Encyclopedia data is of relatively high quality, this embodiment uses Baidu Encyclopedia data as the data source for the knowledge text data. Furthermore, because the abstract portion of Baidu Encyclopedia data has a higher knowledge density than its main text, this embodiment uses abstract text from Baidu Encyclopedia to construct the knowledge text data. After the original corpus is constructed, it relies on Baidu Encyclopedia's classification of abstract entries or constructs its own domain keywords to extract original corpus data related to that domain. For example, for the medical field, it can determine whether the abstract text is related to medical based on existing Baidu Encyclopedia categories of abstract entries or by constructing a common medical terminology list. Through this step, original corpus data related to the application domain can be obtained. Based on the domain-specific original corpus, the jieba tool is used to segment the original corpus, obtain the words and their corresponding frequencies, and filter based on the frequency of occurrence. The specific steps are as follows:

[0068] 1) Calculate the initial frequency threshold, and then filter out domain terms that appear with a frequency lower than the threshold. The calculation method for the initial frequency threshold is as follows:

[0069]

[0070] In the formula, L is the number of characters in the original corpus, N is the number of sentences in the original corpus, aver_l(term) is the average vocabulary length of the public vocabulary, and ρ is the average weight of the estimated vocabulary in the whole sentence, which is 0.2 by default.

[0071] 2) Construct a stop word list, mainly containing prepositions, pronouns, conjunctions, etc. without domain information. First, words with a length of 1 are defaulted to stop words. Then, the stop word list used in this example is constructed based on three Chinese stop word lists: the Chinese stop word list, the Harbin Institute of Technology stop word list, and the Baidu stop word list. The constructed stop word list is then used to further filter the domain word list to improve the domain knowledge density of the word list.

[0072] 3) Assess the size of the domain vocabulary. The specific assessment method is as follows:

[0073]

[0074] In the formula: L is the number of characters in the original corpus, T is the vocabulary set, M is the vocabulary size, F is the frequency of word occurrence, and len(T) = ... i Let be the length of the i-th word in the vocabulary, and θ be the expected average weight of the vocabulary in the entire sentence, which defaults to 0.15. When the calculated value is greater than 1, it means that the proportion of occurrence of the constructed domain vocabulary in the entire corpus is higher than θ, that is, the vocabulary size is appropriate. If it is less than 1, then the frequency threshold needs to be reduced and the stop word screening needs to continue. This process needs to continue until V(L,T,M,F)>1 is satisfied.

[0075] In step S2, refer to Figure 1 This embodiment designs a method for constructing domain knowledge based on ChatGPT to improve the quality and quantity of domain knowledge and promote the subsequent model training process. First, it is necessary to filter domain words with high domain knowledge density from the vocabulary obtained in step S1. The main method is to construct a reasonable interaction paradigm to interact with ChatGPT, such as the paradigm used in this example: I have several words, and you need to classify the words I provide according to their relevance to a certain domain, specifically into three categories: highly relevant, moderately relevant, and irrelevant. Through this interaction paradigm, ChatGPT can obtain the domain relevance of the provided words and filter out the highly relevant words to form a new domain vocabulary with high-density domain knowledge. This domain vocabulary is the preliminary form of domain knowledge obtained in this example and can be used as the masking unit in the full-word masking task, that is, using global context information to predict the domain words. This helps the model learn the meaning, usage, and relationships of words in context, thereby promoting the model's adaptation to the specific domain context.

[0076] Although the domain vocabulary obtained in the previous step contains rich domain features, it lacks information on the differentiation between domain vocabulary. To acquire domain knowledge with high discriminative power, this example uses the paradigm "Assuming you are an expert in a certain field, please classify the vocabulary I provide into sub-fields within that field, with a feedback format of [vocabulary:category]" to interact with ChatGPT and obtain preliminary classification information for the domain vocabulary. This step yields highly discriminative classification knowledge, but because the interaction paradigm does not constrain ChatGPT beyond the domain, the resulting classification knowledge is relatively scattered. That is, the data under each category is sparse, and there is significant overlap between categories. Therefore, manual integration of the obtained domain categories is necessary to combine several categories into a new category, reducing the sparsity of data within categories and the overlap between categories. This step yields a new, less overlapping, and more comprehensive category. This example then uses this category to construct a constrained interaction paradigm: "Assuming you are an expert in a certain field, I will provide you with several vocabulary words and category ranges for that field. You need to classify the provided vocabulary according to the category ranges I provide, with a return format of [vocabulary:category]." This paradigm is used to interact with ChatGPT to obtain the final vocabulary classification knowledge.

[0077] In step S3, refer to Figure 2 This embodiment designs a knowledge fusion method based on multi-task learning, mainly including three training tasks: full-word masking, word comparison learning, and word classification. The full-word masking task utilizes the domain vocabulary obtained in step S2. Based on the random masking task, the masking unit is modified to domain vocabulary. That is, if a word contained in the domain vocabulary exists in the current sentence, then the word is masked at a masking rate of 15%. If the masking rate is less than 15% after masking the word, then random words are masked according to random masking until the masking rate reaches 15%. The masked words or characters are predicted using the context information in the sentence, thereby promoting the model's adaptation to the context of a specific domain.

[0078] The word contrastive learning task, building upon the full-word masking task, uses contrastive learning to constrain the text representation learned by the model. Specifically, the currently masked word can be predicted as multiple words. To constrain the range of predicted words and improve the model's robustness, this example uses the ground truth labels as positive samples at the masked locations and other masked words from the same batch as negative samples of the current masked word. The loss function for this task is calculated as follows:

[0079]

[0080] In the formula: t is [t1, t2, t3, ..., t N], N represents the number of [MASK] in the same training batch, f(t) i ) represents the text representation vector of the i-th [MASK] represented by the model; t′ is [t′1, t′2, t′3, ..., t′] N ],f(t′ i f(t) represents the text representation vector of the true label corresponding to the i-th [MASK] through the model, i.e., f(t) i The positive sample vectors are represented by h1; τ is a hyperparameter that is set, with a default value of 0.01; sim(h1,h2) calculates the cosine similarity between the two vectors h1 and h2, and the calculation formula is as follows:

[0081]

[0082] This contrastive learning task not only constrains the training process of the full-word masking task, but also helps the model gradually learn the semantic similarity and semantic relevance between domain words during the training process, thereby enhancing the model's domain text understanding ability.

[0083] While the full-word masking task and contrastive learning task utilize more domain information compared to traditional tasks, they only leverage information about the broad domain categories. This means they only use the application domain to constrain the model's training, without utilizing internal domain information. Therefore, this example utilizes the internal domain classification information obtained in step S2 to constrain the model's training. Building upon the two tasks mentioned above, it simultaneously provides the classification category information of the masked word within the domain during training. This information further constrains the model's training process and expands the discriminative power between domain words. The specific calculation method for the loss function of the classification task is as follows:

[0084]

[0085] In the formula, C represents the number of categories, and y represents the true label vector of the current data, with a dimension of 1×C. i This represents the true label value of the i-th dimension of the data. This represents the predicted label vector of the current data, that is, the classification vector of the data after passing through the model and classification layer. This represents the predicted label value of the i-th dimension of the data. This loss function can further constrain the model, improve the model's feature discrimination ability in this field, and thus enhance the model's performance in this field.

[0086] Through the above three training tasks, the domain knowledge extracted in step S2 can be well integrated into the model, improving the model's domain knowledge understanding and reasoning ability. At the same time, the domain knowledge is used to constrain the model's training process, thereby improving the model's robustness.

[0087] In step S3, the formula for calculating the total loss function of the training framework is as follows:

[0088] L overall =L WWM +L CL +L WC

[0089] First, obtain the full-word masking loss L from the full-word masking task. WWM This task helps the model learn the meaning, usage, and relationships of words in context, thereby enabling the model to adapt to specific domain contexts; subsequently, the contrastive learning loss L is obtained from the word contrastive learning task. CL This task can gradually help the model learn the semantic similarity and contextual relevance between words during training, thereby improving its ability to understand text; finally, the classification loss L is obtained from the word classification task. WC This loss function provides additional contextual understanding and domain knowledge, thereby improving the model's ability to distinguish different categories in a specific domain.

[0090] Table 1 is a comparison table of the classification task results of this invention on test data in multiple fields.

[0091] Table 1

[0092]

[0093] Table 2 is a comparison table of the retrieval results of the present invention on test data in multiple fields.

[0094] Table 2

[0095]

[0096] Table 1 above shows the evaluation of the framework proposed in this invention compared to other existing methods on text classification tasks in four domains, with the F1 score as the evaluation metric. Table 2 shows the evaluation of the framework proposed in this invention compared to other existing methods on text retrieval tasks in four domains, with the MRR score as the evaluation metric. These two tables demonstrate that the knowledge fusion framework proposed in this invention outperforms other existing methods on multiple tasks across multiple domains. Furthermore, they prove the effectiveness of the knowledge extracted by the knowledge extraction framework of this invention, as the model trained using this knowledge is superior to the baseline model.

[0097] In summary, the preferred embodiments of this invention propose a domain knowledge extraction method based on ChatGPT and a domain knowledge fusion method based on multi-task learning for the domain text classification problem in a specific domain. The method uses a reasonable paradigm construction process to constrain ChatGPT to generate high-quality, fixed-format domain knowledge, and utilizes multiple training tasks to mutually train and constrain each other, thereby improving the model's domain knowledge understanding and reasoning ability, while also improving the model's robustness and enhancing its performance in domain classification tasks.

[0098] like Figure 3 As shown, this embodiment discloses a knowledge extraction and knowledge fusion system based on domain text classification, used to execute the above method embodiment, which includes the following modules:

[0099] Dataset building module: Constructs domain knowledge text datasets;

[0100] Extraction module: Constructs a domain knowledge extraction framework based on ChatGPT to extract domain knowledge contained in domain knowledge text datasets;

[0101] Fusion Module: Constructs a multi-task learning knowledge fusion framework based on contrastive learning and word classification, which integrates the acquired domain knowledge with the deep language model.

[0102] Other aspects of this embodiment can be found in the above method embodiments.

[0103] The above embodiments are merely preferred embodiments and explanations of the technical principles employed in this invention. Those skilled in the art can make other additions or modifications to the described specific embodiments, or substitute them with other methods. Therefore, although the objectives and advantages of this invention are made clearer through the above embodiments, this invention is not limited to the above embodiments. Thus, this specification should not be construed as limiting the invention.

Claims

1. A method for knowledge extraction and knowledge fusion based on domain text classification, characterized in that, Comprise the following steps: S1. Construct a domain knowledge text dataset; S2. Construct a domain knowledge extraction framework based on ChatGPT, extract the domain knowledge contained in the domain knowledge text dataset in step S1; S3. Construct a multi-task learning knowledge fusion framework based on contrastive learning and word classification, and fuse the domain knowledge obtained in step S2 with the deep language model; In step S3, according to the format of the domain knowledge obtained in step S2, a multi-task learning knowledge fusion framework based on contrastive learning and word classification is constructed, which includes three learning tasks: full word mask task, word contrastive learning task and word classification task; The mask unit of the random mask task is changed from a word to a word in the full word mask task; In step S3, the word contrastive learning task is as follows: On the basis of the full word mask task, give the sentence A{x1, x2, x3, x4, x5, x6, x7}, mask x4 and x5 to get sentence A1{x1, x2, x3, [MASK], x6, x7}, and the model output vector of the word set{x4, x5} is taken as the positive sample of the corresponding mask output vector of the A1 sentence, and other words in the same batch data are taken as the negative sample of the corresponding mask output vector of the A1 sentence; Wherein, the calculation formula of the contrastive learning loss function is as follows: In the formula: is , represents the number of [MASK] in the same batch of training, represents the text representation vector of the i-th [MASK] through the model; is , represents the text representation vector of the real label corresponding to the i-th [MASK] through the model, that is the positive sample vector representation of is a set hyperparameter; represents calculating and the cosine similarity of the two vectors, and the calculation formula is as follows: ； In step S3, the word classification task is as follows: On the basis of the full word mask task, give the sentence A{x1, x2, x3, x4, x5, x6, x7}, mask x4 and x5 to get sentence A1{x1, x2, x3, [MASK], x6, x7}, and the output vector of [MASK] is taken as the input of the classification layer, and the classification category information of{x4, x5} obtained in step S2 is taken as the classification label to calculate the word classification task loss; Wherein, the calculation formula of the word classification task loss function is as follows: In the formula, represents the number of categories of classification, represents the true label vector of the current data, with a dimension of 1 x , represents the true label value of the i-th dimension of the data, represents the predicted label vector of the current data, that is, the classification vector of the data after passing through the model and the classification layer, represents the predicted label value of the i-th dimension of the data.

2. The knowledge extraction and fusion method based on domain text classification according to claim 1, wherein, In step S1, the construction of the domain knowledge text dataset includes the following steps: S1.1, original corpus construction, using the word and the corresponding abstract text as the source of the original corpus; S1.2, original corpus classification, divide the categories of abstract text according to the category definition of word, and perform category merging; S1.3, original word table construction, segment the obtained knowledge text, and count the words appearing more than the set frequency to construct the original word table.

3. The method for knowledge extraction and knowledge fusion based on domain text classification according to claim 2, wherein, In step S1.2, according to the required domain, construct domain common keywords and common categories, and extract domain related data text from the original unclassified data using common keywords and common categories.

4. The knowledge extraction and fusion method based on domain text classification according to claim 2, characterized in that, In step S1.3, in order to obtain the original word table while screening stop words, the appearance frequency threshold and the stop word table need to be set; First, calculate the initial appearance frequency threshold, and screen the word table according to the initial appearance frequency threshold; Then construct the stop word table, including the default length of 1 word as stop word and the collection and construction of stop word table; Finally, the original word list is filtered by using the stop word list, and according to the size of the filtered word list, it is determined whether the appearance frequency threshold needs to be reduced to increase the size of the original word list. If the size of the word list meets the requirements, the construction is completed, otherwise the appearance frequency threshold is reduced to increase the size of the original word list; wherein the calculation formula of the initial appearance frequency threshold is as follows: The calculation formula of the evaluation criterion of the word list size is as follows: where: is the number of original corpus characters, is the number of original corpus sentences, is the average vocabulary length of the disclosed vocabulary, is the estimated average proportion weight of the vocabulary in the whole sentence, is the vocabulary set, is the vocabulary size, is the frequency of the word, is the length of the word in the vocabulary, is the expected average proportion weight of the vocabulary in the whole sentence.

5. The method for knowledge extraction and knowledge fusion based on domain text classification according to claim 1, wherein, In step S2, knowledge extraction is specifically as follows: S2.1, on the basis of the word list constructed in step S1, the word list is filtered by using ChatGPT, and the highly relevant words in the field are filtered out; S2.2, on the basis of the field word list filtered out in step S2.1, the field word list is subdivided and classified in the field by using ChatGPT; S2.3, the classified categories are merged and integrated; S2.4, the field classification categories obtained in step S2.3 are used to constrain ChatGPT, and the field word list is reclassified by using the constraint.

6. The method for knowledge extraction and knowledge fusion based on domain text classification according to claim 1, wherein, In step S3, the calculation formula of the total loss function of the multi-task learning knowledge fusion framework is as follows: First, the full word mask loss is obtained from the full word mask task , then the contrast learning loss is obtained from the word contrast learning task , and finally the classification loss is obtained from the word classification task .

7. A system for knowledge extraction and knowledge fusion based on domain text classification for performing the method according to any one of claims 1 to 6, characterized in that The system includes the following modules: Data set construction module: construct a field knowledge text data set; Extraction module: construct a field knowledge extraction framework based on ChatGPT, and extract the field knowledge contained in the field knowledge text data set; Fusion module: construct a multi-task learning knowledge fusion framework based on contrast learning and word classification, and fuse the obtained field knowledge with the deep language model.

Images