Domain adaptive text classification method and system based on knowledge base

Abstract

The invention discloses a domain adaptive text classification method and system based on a knowledge base, and relates to the technical field of artificial intelligence, and the method comprises the steps: constructing a triple between a pivot word and a non-pivot word according to a first target domain untagged corpus, and then constructing a knowledge graph; training a neural network model according to the source domain annotation data, and optimizing the neural network model by using the source domain annotation data, the source domain unannotated data and the second target domain unannotated data; determining a triple corresponding to the pivot word of the input text in the knowledge graph, and inserting the triple into the input text to obtain a knowledge injection text; updating a memory bank by using pseudo tags of the source domain annotation data, the source domain unannotated data and the second target domain unannotated data, so that the neural network model dynamically learns pivot words; and classifying the knowledge injection text by using the neural network model. According to the method, the relationship between the pivot word and the non-pivot word can be modeled through the knowledge graph, so that the accuracy and efficiency of cross-domain adaptive text classification are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a domain adaptation text classification method and system based on a knowledge base. Background Art

[0002] The essence of the knowledge base is a pre-trained large language model. With the wide application of deep learning technology in the field of natural language processing, general pre-trained large language models have shown excellent performance in text classification tasks. However, these models are often difficult to directly migrate to new application fields without additional mainly due to the significant offset in the data distribution in different fields, which is called the domain discrepancy problem. For example, the signal distribution and semantic relationship in product reviews and paper texts are significantly different. The pre-trained model has insufficient understanding of the vocabulary in the data distribution of the new field, resulting in a decline in generalization performance.

[0003] Currently, the methods for domain adaptation using unlabeled data can be roughly divided into two categories: task-independent methods and pivot word methods. Task-independent methods generally ignore the inherent correlation between cross-domain vocabulary, while this correlation contains rich semantic information in natural language processing tasks. In contrast, the pivot word method uses some domain-independent vocabulary (pivot words) to build a bridge between different domains, and learns domain-invariant feature representations through the relationship between pivot words and non-pivot words, so it is more suitable for natural language processing tasks. However, the problems of the pivot word method are that the selection of pivot words and how to model the relationship between pivot words and non-pivot words need to be improved. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a domain adaptation text classification method and system based on a knowledge base to improve the accuracy and efficiency of cross-domain adaptive text classification.

[0005] To achieve the above object, on the one hand, an embodiment of this application proposes a domain adaptation text classification method based on a knowledge base, and the method includes the following steps:

[0006] Construct a triple between pivot words and non-pivot words according to the unlabeled corpus in the first target domain, and construct a knowledge graph according to the triple;

[0007] Train a neural network model according to the labeled data in the source domain, and optimize the neural network model using the labeled data in the source domain, the unlabeled data in the source domain, and the unlabeled data in the second target domain;

[0008] Determine the triple corresponding to the pivot word of the input text in the knowledge graph as the target triple;

[0009] Insert the target triple into the input text to obtain a knowledge-injected text;

[0010] Update the memory bank using the pseudo-labels of the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data, so that the neural network model dynamically learns pivot words;

[0011] Use the neural network model to classify the knowledge injection text.

[0012] In some embodiments, constructing triples between pivot words and non-pivot words based on the first target domain unlabeled corpus includes the following steps:

[0013] On the first target domain unlabeled corpus, use the BERT model to obtain the word-level attention matrix of each sentence;

[0014] Preprocess the word-level attention matrix to obtain a single-head attention matrix;

[0015] According to the single-head attention matrix, determine the first non-pivot word with the highest attention score and the second non-pivot word with the second highest attention score for each pivot word in the sentence;

[0016] The first non-pivot word, the second non-pivot word, and the corresponding pivot word form one of the triples;

[0017] Constructing a knowledge graph according to the triples includes the following steps:

[0018] Delete the triples whose sum of attention scores is lower than a set threshold;

[0019] Construct the knowledge graph according to the remaining triples.

[0020] In some embodiments, training a neural network model according to the source domain labeled data includes the following steps:

[0021] Train a knowledge injection P-BERT classification model as the neural network model according to the source domain labeled data;

[0022] Optimizing the neural network model using the source domain labeled data, source domain unlabeled data, and second target domain unlabeled data includes the following steps:

[0023] Use the source domain labeled data, source domain unlabeled data, and second target domain unlabeled data to optimize the knowledge injection P-BERT classification model; wherein, the knowledge injection P-BERT classification model includes a knowledge injection module and a BERT encoder; the objective function of the knowledge injection P-BERT classification model is cross-entropy loss, and the objective function is:

[0024] L = -∑y log(P-Bert(x)), where x,y ∈ L_S;

[0025] Here, L_S represents the source domain labeled data, (x, y) is a source domain labeled sample; P-BERT(x) is the predicted value of the knowledge injection P-BERT classification model.

[0026] In some embodiments, inserting the target triple into the input text to obtain the knowledge injection text includes the following steps:

[0027] Map the input text into a tree structure and represent the tree structure as a sequence;

[0028] Introduce a structured position for each pivot word in the sequence;

[0029] Embed the target triple at the structured position to obtain the knowledge injection text.

[0030] In some embodiments, updating the memory bank using the source domain labeled data, the source domain unlabeled data, and the pseudo-labels of the second target domain unlabeled data to enable the neural network model to dynamically learn pivot words includes the following steps:

[0031] Define the polarity score of word w in domain D:

[0032] p(w,D) = (|{l = 1|l ∈ b(w,D)}| - |{l = -1|l ∈ b(w,D)}|) / |b(w,D);

[0033] Here, b(w,D) is the label set of word w in domain D, and the purpose of p(w,D) is to measure the bias of word w towards a certain category in domain D;

[0034] Define the behavior score of word w in domain pair (S,T):

[0035] score(w,S,T) = |p(w,S) + p(w,T)| / 2;

[0036] Here, score(w,S,T) evaluates the inter-domain transferability through the consistency of the source domain and the target domain;

[0037] Iteratively update the memory bank using pseudo-labels and true labels:

[0038] p(w,D) ← α*p(w,D) + (1 - α)*l;

[0039] Here, the iterative update of the memory bank is used to improve the accuracy of evaluating the polarity of word w.

[0040] In some embodiments, classifying the knowledge-injected text by using the neural network model includes the following steps:

[0041] Using a knowledge-aware Transformer network as the neural network model to learn the feature representation of the knowledge-injected text;

[0042] Classifying the feature representation to classify the knowledge-injected text.

[0043] In some embodiments, the method further includes at least one of the following steps:

[0044] Pre-selecting pivot words by a statistical information method or a clustering algorithm;

[0045] Alternatively, using a pre-trained language model to extract word vectors and constructing the knowledge graph by using the similarity of the word vectors.

[0046] To achieve the above object, another aspect of the embodiments of the present application provides a domain adaptation text classification system based on a knowledge base, and the system includes:

[0047] A knowledge graph construction module, configured to construct triples between pivot words and non-pivot words according to unlabeled corpus in a first target domain, and construct a knowledge graph according to the triples;

[0048] A model training and optimization module, configured to train a neural network model according to source domain labeled data, and optimize the neural network model by using the source domain labeled data, source domain unlabeled data, and second target domain unlabeled data;

[0049] A triple determination module, configured to determine the triple corresponding to the pivot word of the input text in the knowledge graph as a target triple;

[0050] A knowledge injection module, configured to insert the target triple into the input text to obtain a knowledge-injected text;

[0051] A pivot word acquisition module, configured to update a memory bank by using the pseudo-labels of the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data, so that the neural network model dynamically learns pivot words;

[0052] A text classification module, configured to classify the knowledge-injected text by using the neural network model.

[0053] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, and the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0054] To achieve the above object, on the other hand, an embodiment of the present application proposes a computer-readable storage medium storing a computer program which, when executed by a processor, implements the above method.

[0055] The embodiments of the present application at least include the following beneficial effects:

[0056] The solution of the present application includes: constructing triples between pivot words and non-pivot words according to unannotated corpus in the first target domain, and constructing a knowledge graph according to the triples; training a neural network model according to source domain annotation data, and optimizing the neural network model using source domain annotation data, source domain unannotated data, and second target domain unannotated data; determining the triples corresponding to the pivot words of the input text in the knowledge graph as target triples; inserting the target triples into the input text to obtain a knowledge-injected text; updating the memory bank using the pseudo-labels of source domain annotation data, source domain unannotated data, and second target domain unannotated data, so that the neural network model dynamically learns pivot words; and classifying the knowledge-injected text using the neural network model. By constructing a structured knowledge graph in the target domain, the present application can explicitly model the relationship between pivot words and non-pivot words, avoid introducing noise by simple co-occurrence relationships, can more accurately express the semantic connection between words, and thus improve the accuracy and efficiency of cross-domain adaptive text classification. Description of the Drawings

[0057] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings.

[0058] Figure 1 It is a schematic flowchart of a domain adaptive text classification method based on a knowledge base provided by an embodiment of the present application;

[0059] Figure 2 It is an example flowchart of a domain adaptive text classification method based on a knowledge base provided by an embodiment of the present application;

[0060] Figure 3 It is a schematic structural diagram of a domain adaptive text classification system based on a knowledge base provided by an embodiment of the present application;

[0061] Figure 4 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0062] To make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0063] It can be understood that the terms "first", "second", etc. used in the present application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if", "when" as used herein may be interpreted as "when...", "while...", or "in response to determining".

[0064] The terms "at least one", "a plurality", "each", "any one", etc. used in the present application, at least one includes one, two or more than two, a plurality includes two or more than two, each refers to each of the corresponding plurality, and any one refers to any one of the plurality.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0066] Before the embodiments of the present application are described in detail, some related technologies involved in the embodiments of the present application will be described first, as follows:

[0067] Related technology 1 proposes to add two technical means on the basis of a pre-trained BERT model to achieve domain adaptation of the model. The first is to use an adversarial training framework and add a gradient reversal layer at the end of the BERT encoder, so that the encoder learns feature representations that are suitable for the data distributions of both the source domain and the target domain, in order to reduce the domain difference. The second is to further fine-tune the pre-trained BERT using the data of the target domain, and correct the semantic understanding deviation of the model in the target domain through the supervision signal of the target domain. The combination of these two technologies can enable the model to better model the data of the target domain.

[0068] Disadvantages of Related Technology 1: It is necessary to introduce additional adversarial training to adapt to the new domain, which increases the computational complexity of model training. At the same time, the learned feature representations do not have a sufficient understanding of the consistency of the lexical semantic attributes of the source domain and the target domain, reducing the generalization ability of the model.

[0069] Related Technology 2 specifically constructs an external knowledge graph that contains the shared entity relationship knowledge of the source domain and the target domain. Its technical process is as follows: First, use various knowledge sources to construct a cross-domain knowledge graph; then design a knowledge-aware module to insert knowledge triples into the text representation; next, based on the text representation with inserted knowledge, train the Transformers encoder to obtain a knowledge-aware text representation; finally, add a classification layer on the basis of the encoder representation for fine-tuning to obtain the final domain adaptation sentiment classification model. This method makes up for the difference in lexical distribution between the source domain and the target domain by introducing an external knowledge graph.

[0070] Disadvantages of Related Technology 2: It is overly dependent on the additionally constructed knowledge graph. It needs to specifically construct a new knowledge graph for different source domains and target domains and cannot be applied to scenarios without a pre-constructed knowledge graph, restricting its scope of application. Moreover, the process of constructing the knowledge graph in this method is time-consuming and laborious, which is not conducive to fast and efficient domain adaptation.

[0071] Based on this, the technical problems to be solved by this application include:

[0072] 1) Introducing an additional adversarial training mechanism will increase the training calculation amount and complexity of the model. This problem will lead to a reduction in the training efficiency of the additional adaptation model and it is difficult to apply it to scenarios that require fast and efficient training. Additionally, relying too much on adversarial training to learn inter-domain features will also affect the consistency of the feature learned by the model in understanding the semantics of the source domain and the target domain, reducing the generalization ability of the model.

[0073] 2) The problem of dependence on the knowledge graph source. Related Technology 2 needs to manually construct a dedicated knowledge graph for each domain and cannot be applied to scenarios without a pre-constructed knowledge graph. This severely restricts the scope of application of the method and also increases the resource consumption in the application. Moreover, manually constructing a knowledge graph is very time-consuming and laborious, making it difficult for this method to be quickly iterated and updated.

[0074] Therefore, the embodiments of the present application provide. The technical solutions of the present application include: constructing triples between pivot words and non-pivot words according to the unlabeled corpus in the first target domain, and constructing a knowledge graph according to the triples; training a neural network model according to the source domain labeled data, and optimizing the neural network model using the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data; determining the triples corresponding to the pivot words of the input text in the knowledge graph as target triples; inserting the target triples into the input text to obtain a knowledge-injected text; updating the memory bank using the pseudo-labels of the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data, so that the neural network model dynamically learns pivot words; and classifying the knowledge-injected text using the neural network model. By constructing a structured knowledge graph in the target domain, the present application can explicitly model the relationship between pivot words and non-pivot words, avoid introducing noise due to simple co-occurrence relationships, more accurately express the semantic connections between words, and thus improve the accuracy and efficiency of cross-domain adaptive text classification.

[0075] The embodiments of the present application provide a domain adaptive text classification method and system based on a knowledge base, which relates to the field of artificial intelligence technology. The domain adaptive text classification method and system provided by the embodiments of the present application can be applied to a terminal, or can be applied to a server, or can also be software running on a terminal or a server. In some embodiments, the terminal may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing a domain adaptive text classification method based on a knowledge base, etc., but is not limited to the above forms.

[0076] This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are executed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] Referring to Figure 1 , an embodiment of this application provides a domain adaptation text classification method based on a knowledge base. This method may include but is not limited to steps S100 to S150, specifically as follows:

[0078] S100: Construct triples between pivot words and non-pivot words based on the unlabeled corpus of the first target domain, and construct a knowledge graph according to the triples.

[0079] Further, constructing triples between pivot words and non-pivot words in S100 based on the unlabeled corpus of the first target domain includes the following steps S101 to S104:

[0080] S101: Use the BERT model on each sentence in the unlabeled corpus of the first target domain to obtain the word-level attention matrix of the sentence;

[0081] S102: Preprocess the word-level attention matrix to obtain a single-head attention matrix;

[0082] S103: Determine the first non-pivot word with the highest attention score and the second non-pivot word with the second-highest attention score for each pivot word in the sentence according to the single-head attention matrix;

[0083] S104: The first non-pivot word, the second non-pivot word, and the corresponding pivot word form one of the triples.

[0084] Further, in S100, it includes the following steps S105 to S106:

[0085] S105: Delete the triples whose sum of attention scores is lower than the set threshold;

[0086] S106: Construct the knowledge graph according to the remaining triples.

[0087] S110: Train a neural network model based on the source domain labeled data, and optimize the neural network model using the source domain labeled data, source domain unlabeled data, and second target domain unlabeled data.

[0088] Further, training the neural network model according to the source domain labeled data in S110 includes the following steps:

[0089] S111: Train a knowledge injection P-BERT classification model according to the source domain labeled data as the neural network model.

[0090] Further, optimizing the neural network model using the source domain labeled data, source domain unlabeled data, and second target domain unlabeled data in S110 includes the following step S112:

[0091] S112: Optimize the knowledge injection P-BERT classification model using the source domain labeled data, source domain unlabeled data, and second target domain unlabeled data; wherein, the knowledge injection P-BERT classification model includes a knowledge injection module and a BERT encoder; the objective function of the knowledge injection P-BERT classification model is cross-entropy loss, and the objective function is:

[0092] L = -∑y log(P-Bert(x)), x, y ∈ L_S;

[0093] wherein, L_S represents the source domain labeled data, (x, y) is a source domain labeled sample; P-BERT(x) is the predicted value of the knowledge injection P-BERT classification model.

[0094] S120: Determine the triple corresponding to the pivot word of the input text in the knowledge graph as the target triple.

[0095] S130: Insert the target triple into the input text to obtain a knowledge injection text.

[0096] Further, S130 may include the following steps S131 to S133:

[0097] S131: Map the input text into a tree structure and represent the tree structure in a sequence.

[0098] S132: Introduce a structured position for each pivot word in the sequence.

[0099] S133: Embed the target triple at the structured position to obtain the knowledge injection text.

[0100] S140: Update the memory bank using the pseudo-labels of the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data, so that the neural network model dynamically learns pivot words.

[0101] Further, S140 may include the following steps:

[0102] S141: Define the polarity score of word w in domain D:

[0103] p(w,D) = (|{l = 1|l ∈ b(w,D)}| - |{l = -1|l ∈ b(w,D)}|) / |b(w,D);

[0104] where b(w,D) is the label set of word w in domain D, and the purpose of p(w,D) is to measure the bias of word w towards a certain category in domain D;

[0105] S142: Define the behavior score of word w in domain pair (S,T):

[0106] score(w,S,T) = |p(w,S) + p(w,T)| / 2;

[0107] where score(w,S,T) evaluates the transferability between domains through the consistency of the source domain and the target domain;

[0108] S143: Iteratively update the memory bank using pseudo-labels and true labels:

[0109] p(w,D) ← α * p(w,D) + (1 - α) * l;

[0110] where the iterative update of the memory bank is used to improve the accuracy of evaluating the polarity of word w.

[0111] S150: Classify the knowledge injection text using the neural network model.

[0112] Further, S150 may include the following steps S151 to S152:

[0113] S151: Use a knowledge-aware Transformer network as the neural network model to learn the feature representation of the knowledge injection text;

[0114] S152: Classify the feature representation to classify the knowledge injection text.

[0115] Further, the embodiments of the present application may further include at least one of the steps S161 or S162:

[0116] S161: Pre-select pivot words through statistical information methods or clustering algorithms;

[0117] S162: Extract word vectors using a pre-trained language model, and construct the knowledge graph using the similarity of the word vectors.

[0118] Next, specific application examples will be combined to introduce and explain the solutions of the embodiments of this application in detail.

[0119] This embodiment proposes a domain adaptation text classification method based on a knowledge base to solve the problem of weak generalization ability of current pre-trained large language models when facing information in different domains. This method may include the following steps: First, construct a structured knowledge graph on the unlabeled corpus in the target domain to express the relationship between pivot words and non-pivot words in the form of triples; then, design a knowledge injection mechanism associated with pivot words and inject relevant triples obtained from the knowledge graph into the source domain data; next, use a knowledge-aware Transformer network to learn the feature representation of the knowledge-injected text and perform classification fine-tuning to obtain the final model. Compared with the prior art, the present invention avoids manual annotation by constructing a target domain knowledge graph, effectively utilizes the relationship between vocabulary for transfer through knowledge injection, and improves adaptability by dynamically learning pivot words.

[0120] Exemplarily, Figure 2 It is an example flowchart of a domain adaptation text classification method based on a knowledge base.

[0121] Specifically, this embodiment may include the following solutions:

[0122] (1) Input source domain labeled data and target domain unlabeled data;

[0123] (2) On the first target domain unlabeled corpus, construct a structured knowledge graph to model the relationship between pivot words and non-pivot words;

[0124] (3) Train a neural network model on the source domain labeled data, and simultaneously optimize the model using the source domain labeled data, source domain unlabeled data, and second target domain unlabeled data; calculate the confidence of the fact triples in the knowledge graph and filter out the fact triples with low confidence;

[0125] (4) At the input end of the model, inject the fact triples in the source domain into the target domain text for knowledge injection;

[0126] (5) Input the knowledge-injected input data into a knowledge-aware Transformer encoder to learn the knowledge-aware text feature representation;

[0127] (6) Based on the encoder features, add a classification layer for fine-tuning to obtain the final cross-domain text classification model.

[0128] The relevant algorithm formulas used in this embodiment are introduced as follows:

[0129] Model processing flow:

[0130] A. Construct a knowledge graph and establish the relationship between modeling pivot words and non-pivot words:

[0131] On the unlabeled corpus U_T in the target domain, for each sentence s, use the BERT model to obtain the word-level attention matrix A of the sentence. Preprocess A to obtain the single-head attention matrix M. For each pivot word p in the sentence, find the word w1 with the highest attention score and the second-highest word w2, and form a knowledge triple (w1, w2, p). Filter according to the sum of the attention scores of the triples to construct the knowledge graph KG.

[0132] Calculate the confidence score:

[0133] score = A[p][w1] + A[p][w1];

[0134] Filter the knowledge triples with low confidence to construct the knowledge graph KG.

[0135] B. Knowledge injection:

[0136] Use the true labels of the labeled source data L_S, the pseudo-labels of the unlabeled source data U_S, and the pseudo-labels of the unlabeled target domain data U_T to jointly update the memory bank and dynamically learn pivot words (Pivot Learning):

[0137] TextInput = InsertKGFacts(x, p, (w1, w2, p)).

[0138] Among them, the positional embedding is used to restore the tree structure, and the positional embedding index of each token is specified as its depth in the tree.

[0139] C. Knowledge graph-guided model optimization:

[0140] Use the true labels of L_S, the pseudo-labels of U_S, and the pseudo-labels of U_T to update the memory bank and dynamically learn pivot words (Pivot Learning):

[0141] 1) Define the polarity score of word w in domain D:

[0142] p(w, D) = (|{l = 1|l ∈ b(w, D)}| - |{l = -1|l ∈ b(w, D)}|) / |b(w, D);

[0143] Among them, b(w, D) is the label set of w in domain D, and the purpose of p(w, D) is to measure the bias of word w towards a certain category in domain D.

[0144] 2) Define the behavior score of word w on the domain pair (S, T):

[0145] p s (w, (S, T)) = |p(w, S) + p(w, T)| / 2;

[0146] p s (w, (S, T)) evaluates the inter-domain transferability through the consistency between the source domain and the target domain.

[0147] 3) During the training process, at each training step, if the prediction confidence of the softmax probability during classification exceeds the threshold, then obtain the pseudo-labels of the unlabeled text. Use the pseudo-labels of the unlabeled source domain inputs and the true labels of the labeled source domain inputs to update the source memory bank, while use the pseudo-labels of the target domain inputs to update the target memory bank. The update is performed in a time-difference manner: for each candidate pivot word, if it is labeled as l ∈ {1, -1} in the text of domain D, then:

[0148] p(w, D) ← α * p(w, D) + (1 - α) * l;

[0149] where α is the update rate.

[0150] 4) At the beginning of each epoch, calculate the absolute average score of the candidate pivot words through the calculation formula of the behavior score, and select the top K words with the highest scores as the pivot words.

[0151] D. Fine-tune the classification model:

[0152] Input TextInput into the Transformer encoder to obtain the representation h, which is predicted by a fully connected layer with softmax, and use the cross-entropy loss as the objective function:

[0153] L = -∑y log(P-BERT(x));

[0154] where (x, y) is the source domain annotation sample, and P-BERT(x) is the model prediction. In the model training, minimizing the objective loss function is used as the training goal.

[0155] In summary, the technical solution of this embodiment may include:

[0156] 1) A method for constructing a structured knowledge graph based on the target domain corpus:

[0157] In this embodiment, the BERT model is used to extract the attention matrix of the target domain corpus, and a structured knowledge graph is constructed according to the matrix. The graph explicitly represents the relationship between the pivot words and non-pivot words in the form of triples. This avoids the knowledge noise caused by simple co-occurrence relationships and can more accurately model the connections between words.

[0158] 2) Knowledge injection mechanism associated with pivot words:

[0159] In this embodiment, triples related to the pivot words are obtained from the knowledge graph and injected into the positions related to the pivot words in the source domain input text. This enables the source domain text to contain target domain vocabulary, allowing the model to learn the feature representations of non-pivot words under supervised conditions and achieve knowledge transfer.

[0160] The beneficial effects brought by the technical solution of this embodiment include:

[0161] 1) By constructing a structured knowledge graph in the target domain in this embodiment, the relationship between pivot words and non-pivot words can be explicitly modeled, avoiding the introduction of noise by simple co-occurrence relationships and more accurately expressing the semantic connections between vocabulary.

[0162] 2) Using the knowledge injection technology associated with pivot words in this embodiment, knowledge related to the target domain can be injected into the source domain data, learning the feature representations of non-pivot words under supervised conditions, effectively using the relationships between vocabulary for transfer, and thus avoiding the introduction of additional adversarial training operations and reducing the computational complexity of model training.

[0163] 3) Dynamically learning pivot words in this embodiment can continuously track the behavior of vocabulary in the source domain and the target domain, and select pivot words that are more beneficial for domain transfer. This is more conducive to improving the domain adaptation ability of the model than predefined static pivot words.

[0164] 4) Since this embodiment directly learns the knowledge graph from the target domain text and avoids manual annotation, this embodiment can be extended to new tasks where there is no predefined knowledge graph, improving the scalability of the method.

[0165] In addition, this embodiment also provides the following alternative solutions:

[0166] 1) Instead of obtaining pivot words in a dynamically learning manner, pivot words are preselected through statistical information methods or clustering algorithms.

[0167] 2) Use a pre-trained language model (such as BERT) to extract word vectors and construct a knowledge graph using the similarity of word vectors instead of using the attention matrix.

[0168] Refer to Figure 3 , this application embodiment also provides a domain adaptive text classification system based on a knowledge base, which can implement the above-mentioned domain adaptive text classification method based on a knowledge base. The system includes:

[0169] A knowledge graph construction module, configured to construct triples between pivot words and non-pivot words according to the unlabeled corpus in the first target domain, and construct a knowledge graph according to the triples;

[0170] A model training and optimization module, configured to train a neural network model according to the source domain labeled data, and optimize the neural network model by using the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data;

[0171] A triple determination module, configured to determine the triple corresponding to the pivot word of the input text in the knowledge graph as the target triple;

[0172] A knowledge injection module, configured to insert the target triple into the input text to obtain a knowledge-injected text;

[0173] A pivot word acquisition module, configured to update the memory bank by using the pseudo-labels of the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data, so that the neural network model dynamically learns pivot words;

[0174] A text classification module, configured to classify the knowledge-injected text by using the neural network model.

[0175] It can be understood that the content in the above method embodiments is applicable to the system embodiments of the present application. The functions specifically implemented by the system embodiments of the present application are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0176] An embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the method of the embodiment of the present application is implemented. The electronic device may be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0177] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those of the method of the present application, and the beneficial effects achieved are also the same as those of the method of the present application.

[0178] Please refer to Figure 4 , Figure 4 which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0179] The processor 401 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0180] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 402 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 402 and are called by the processor 401 to execute the methods in the embodiments of the present application;

[0181] The input / output interface 403 is used to implement information input and output;

[0182] The communication interface 404 is used to implement communication interaction between this device and other devices, and can communicate through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0183] The bus 405 transmits information between various components of the device (such as the processor 401, the memory 402, the input / output interface 403, and the communication interface 404);

[0184] Among them, the processor 401, the memory 402, the input / output interface 403, and the communication interface 404 achieve communication connections with each other inside the device through the bus 405.

[0185] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the present application is implemented.

[0186] It can be understood that the content in the above method embodiments is applicable to the embodiments of this storage medium. The functions specifically implemented by the embodiments of this storage medium are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0187] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0188] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0189] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0190] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0192] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules does not have to be limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products, or devices.

[0193] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist simultaneously. Here, A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the associated objects before and after. "At least one (item) of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0194] In several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of systems or modules can be in electrical, mechanical, or other forms.

[0195] The modules described above as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, in each embodiment of this application, the various functional modules can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0197] When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0198] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, and thus do not limit the scope of the rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall be within the scope of the rights of the embodiments of this application.

Claims

1. A domain adaptation text classification method based on a knowledge base, characterized in that The method includes the following steps: Construct triples between pivot words and non-pivot words based on the unannotated corpus of the first target domain, and construct a knowledge graph according to the triples; Train a neural network model based on the source domain annotation data, and optimize the neural network model using the source domain annotation data, the source domain unannotated data, and the second target domain unannotated data; Determine the triples corresponding to the pivot words of the input text in the knowledge graph as target triples; Insert the target triples into the input text to obtain a knowledge-injected text; Update the memory bank using the pseudo-labels of the source domain annotation data, the source domain unannotated data, and the second target domain unannotated data, so that the neural network model dynamically learns pivot words; Classify the knowledge-injected text using the neural network model.

2. The domain adaptation text classification method based on a knowledge base according to claim 1, characterized in that, The step of constructing triples between pivot words and non-pivot words based on the unannotated corpus of the first target domain includes the following steps: Obtain the word-level attention matrix of each sentence on the unannotated corpus of the first target domain using the BERT model; Preprocess the word-level attention matrix to obtain a single-head attention matrix; Determine the first non-pivot word with the highest attention score and the second non-pivot word with the second highest attention score for each pivot word in the sentence according to the single-head attention matrix; The first non-pivot word, the second non-pivot word, and the corresponding pivot word form a triple; The step of constructing a knowledge graph according to the triples includes the following steps: Delete the triples whose sum of attention scores is lower than a set threshold; Construct the knowledge graph according to the remaining triples.

3. A domain adaptation text classification method based on a knowledge base according to claim 1, characterized in that The step of training a neural network model based on the source domain annotation data includes the following steps: Train a knowledge-injected P-BERT classification model as the neural network model according to the source domain annotation data; The step of optimizing the neural network model using the source domain annotation data, the source domain unannotated data, and the second target domain unannotated data includes the following steps: Optimize the knowledge-injected P-BERT classification model using the source domain annotation data, the source domain unannotated data, and the second target domain unannotated data; wherein, the knowledge-injected P-BERT classification model includes a knowledge injection module and a BERT encoder; the objective function of the knowledge-injected P-BERT classification model is cross-entropy loss, and the objective function is: L = -∑ylog(P-Bert(x)), x, y ∈ L_S; wherein, L_S represents the source domain annotation data, (x, y) is a source domain annotation sample; P-BERT(x) is the predicted value of the knowledge-injected P-BERT classification model.

4. The domain-adaptive text classification method based on a knowledge base according to claim 1, characterized in that: The step of inserting the target triples into the input text to obtain a knowledge-injected text includes the following steps: Map the input text into a tree structure and represent the tree structure in a sequence; Introduce a structured position for each pivot word in the sequence; Embed the target triples at the structured positions to obtain the knowledge-injected text.

5. A domain adaptation text classification method based on a knowledge base according to claim 1, characterized in that, Updating the memory bank using the pseudo-labels of the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data, so that the neural network model dynamically learns pivot words, includes the following steps: Define the polarity score of word w in domain D: p(w,D) = (|{l = 1|l ∈ b(w,D)}| - |{l = -1|l ∈ b(w,D)}|) / |b(w,D); where b(w,D) is the label set of word w in domain D, and the purpose of p(w,D) is to measure the bias of word w towards a certain category in domain D; Define the behavior score of word w in domain pair (S,T): score(w,S,T) = |p(w,S) + p(w,T)| / 2; where score(w,S,T) evaluates the transferability between domains through the consistency of the source domain and the target domain; Iteratively update the memory bank using pseudo-labels and true labels: p(w,D) ← α*p(w,D) + (1 - α)*l; where the iterative update of the memory bank is used to improve the accuracy of evaluating the polarity of word w.

6. The domain adaptation text classification method based on a knowledge base according to claim 1, wherein Classifying the knowledge injection text using the neural network model includes the following steps: Using a knowledge-aware Transformer network as the neural network model to learn the feature representation of the knowledge injection text; Classifying the feature representation to classify the knowledge injection text.

7. A domain adaptation text classification method based on a knowledge base according to any one of claims 1 to 6, characterized in that, The method further includes at least one of the following steps: Pre-selecting pivot words by statistical information method or clustering algorithm; Or, using a pre-trained language model to extract word vectors, and constructing the knowledge graph using the similarity of the word vectors.

8. A domain adaptation text classification system based on a knowledge base, characterized in that, The system includes: A knowledge graph construction module, configured to construct triples between pivot words and non-pivot words according to the first target domain unlabeled corpus, and construct a knowledge graph according to the triples; A model training and optimization module, configured to train a neural network model according to the source domain labeled data, and optimize the neural network model using the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data; A triple determination module, configured to determine the triple corresponding to the pivot word of the input text in the knowledge graph as the target triple; A knowledge injection module, configured to insert the target triple into the input text to obtain a knowledge injection text; A pivot word acquisition module, configured to update the memory bank using the pseudo-labels of the source domain labeled data, the source domain unlabeled data, and the second target domain unlabeled data, so that the neural network model dynamically learns pivot words; A text classification module, configured to classify the knowledge injection text using the neural network model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images