Method for constructing vertical knowledge graph based on transfer learning

Abstract

The application provides a vertical field knowledge graph construction method based on transfer learning, which has the following characteristics and comprises the following steps: step S1, inputting knowledge text into a pre-trained natural language model A to obtain word units; step S2, inputting natural language labels into a pre-trained natural language model B to obtain a feature representation set; step S3, performing dot product calculation on the word units and the feature representation set to obtain a classification result; step S4, inputting existing relationship words filled in a Prompt template into a pre-trained natural language model C to obtain a vector representation; step S5, inputting a classified sentence into a pre-trained natural language model D to obtain an encoding result; step S6, performing similarity calculation on the encoding result and the vector representation to obtain a relationship classification result; and step S7, constructing a tuple according to the word units and the relationship classification result to construct a knowledge graph. In summary, the method has good efficiency and quality for entity extraction in a vertical field, and has good scalability in relationship extraction.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of knowledge graph, and particularly relates to a vertical field knowledge graph construction method based on transfer learning. BACKGROUND

[0002] Knowledge extraction and knowledge representation are two key technologies for constructing a knowledge graph, and their importance is self-evident as upstream tasks. Knowledge extraction, as one of them, mainly solves the problem of how to simply and efficiently extract knowledge information such as entities, relationships, attributes, events, etc. from multiple heterogeneous data sources, and then process these knowledge to form the required tuples of the knowledge graph. In terms of the types of knowledge extracted, there are mainly two key tasks: relation extraction and entity extraction. The data composed of structured, semi-structured, natural language, etc. is processed into the form of triples (entity 1, relationship, entity 2), and this form of data is the basis for constructing a knowledge graph.

[0003] The quality of entity extraction (Named Entity Recognition, NER) also affects the downstream tasks of the knowledge graph, such as the effects of the tasks of question answering, recommendation, etc. The current entity extraction has been relatively mature in the industry, and the most commonly used method is the Bi-LSTM + CRF model.

[0004] Relation extraction (Relation Extraction, RE) has always been a key research content in the field of natural language processing, and as another key technology for constructing a knowledge graph, the purpose is to mine the relationship information between entities in the annotated or unannotated corpus. With the rapid development of neural networks and deep learning in recent years, the relation extraction model has also changed greatly since 2010, and the effect has also greatly exceeded the previous traditional methods.

[0005] At the same time, with the development of deep learning in recent years, various open-source automatic extraction and construction tools for knowledge graphs have emerged in an endless stream, and their appearance has also reduced the difficulty of constructing a knowledge graph. For example, DeepKE is a Chinese knowledge graph construction framework based on deep learning, which supports a few-shot, long-sequence knowledge extraction tool, and supports mainstream natural language processing tasks such as named entity recognition, relation extraction, etc.

[0006] In addition, for the two key tasks of entity extraction and relation extraction for constructing a knowledge graph, there are usually two types of solutions in the academic field. One type is to separate the entity and relation extraction processes and train and optimize them as two separate models. The other type is to combine the two processes and train a model to achieve the two task goals.

[0007] The knowledge graph in the vertical field develops more slowly than the general field. As a professional field knowledge graph, the depth of knowledge should be highlighted, and the data sources are relatively narrow, mainly relying on professional books, patents, models, experience, etc. It is difficult to obtain through automatic tools, and the quality of knowledge is required to be high, and it is difficult to use, and specific field knowledge is required. The common form of vertical field knowledge is mainly composed of text, pictures, videos and some special format file data.

[0008] In the aspect of extracting knowledge in the vertical field, there are two types of extraction methods based on rules and representation. The rule-based method needs to consume a lot of human resources to construct rules, and most of the methods need manual verification and the ability of the model is also limited. The representation-based method uses neural network training model to reduce the participation of artificial, but due to the lack of data in the vertical field, the model may have poor generalization and the training result is difficult to migrate. At the same time, if the abstract mode of the vertical field knowledge changes, it will directly affect the downstream knowledge extraction model, and the influence is also great.

[0009] In summary, in the scene of constructing the knowledge graph in the vertical field, the existing technology cannot balance the quality of knowledge extraction, the efficiency of constructing the knowledge graph and the maintenance of the knowledge graph. SUMMARY

[0010] The present application is to solve the above problems, and aims to provide a vertical field knowledge graph construction method based on transfer learning.

[0011] The application provides a vertical field knowledge graph construction method based on transfer learning, which is used for constructing a vertical field knowledge graph according to predetermined knowledge texts of the vertical field, natural language labels representing entity categories and existing relationship words. The method has the following characteristics, including the following steps: step S1, inputting the knowledge texts into a pre-trained natural language model A to obtain each word element of each sentence in the knowledge texts; step S2, inputting a plurality of natural language labels into a pre-trained natural language model B to obtain feature representations corresponding to each natural language label as a feature representation set; step S3, expanding the word element to the size of the feature representation set, and performing dot product calculation with the feature representation set, marking the word element with the corresponding natural language label according to the feature representation corresponding to the maximum weight obtained by the calculation, as the classification result of the word element; step S4, filling m existing relationship words into a Prompt template to obtain m relationship sentences, and inputting the m relationship sentences into a pre-trained natural language model C to obtain vector representations of the m relationship sentences; step S5, inputting the sentence after marking and classifying the word element into a pre-trained natural language model D for encoding to obtain an encoding result; step S6, performing similarity calculation on the encoding result and the vector representations of the m relationship sentences respectively, and selecting the existing relationship word corresponding to the maximum value of the similarity calculation result as the relationship classification result between the word elements in the sentence; step S7, constructing a tuple according to the word element in the sentence and the corresponding relationship classification result as the knowledge extraction result of the sentence, and constructing the knowledge graph of the vertical field through the knowledge extraction results of all sentences of the knowledge texts, wherein the pre-trained natural language model A, the pre-trained natural language model B, the pre-trained natural language model C and the pre-trained natural language model D are obtained by performing transfer learning on a BERT model according to a training data set.

[0012] In the vertical field knowledge graph construction method based on transfer learning provided by the application, the training data set can also have the following characteristics: the training data set includes vertical field training data and enhanced training data processed by an enhancement algorithm from the training data, and the process of obtaining the pre-trained natural language model A, the pre-trained natural language model B, the pre-trained natural language model C and the pre-trained natural language model D by performing transfer learning on the BERT model according to the training data set includes the following steps: step T1, inputting the training data into the BERT model L and regularizing the output result to obtain a sentence feature representation ; step T2, inputting the enhanced training data into the BERT model T and regularizing the output result to obtain a sentence feature representation ; step T2, inputting the enhanced training data into the BERT model T and regularizing the output result to obtain a sentence feature representation ; step T2, inputting the enhanced training data into the BERT model T and regularizing the output result to obtain a sentence feature representation , the parameters of the BERT model L and the BERT model T are updated respectively through similarity calculation and loss calculation, to obtain the BERT model L and the BERT model T after migration learning, the pre-training natural language model A is any one of the BERT model L and the BERT model T after migration learning, the pre-training natural language model B is the other one of the BERT model L and the BERT model T after migration learning, the pre-training natural language model C is the BERT model T after migration learning, and the pre-training natural language model D is the BERT model L after migration learning.

[0013] In the vertical field knowledge graph construction method based on migration learning provided by the application, the enhancement algorithm can be a back-translation method.

[0014] In the vertical field knowledge graph construction method based on migration learning provided by the application, the BERT model, the BERT model L and the BERT model T can all be pre-training natural language models RoBERTa-wwm-ext.

[0015] In the vertical field knowledge graph construction method based on migration learning provided by the application, after obtaining the pre-training natural language model A and the pre-training natural language model B through migration learning, the pre-training natural language model A and the pre-training natural language model B are trained according to an entity training data set, the entity training data set includes training text, training natural language labels and actual labels of each word in the training text, and includes the following steps: step R1, inputting the training text into the pre-training natural language model A to obtain each word in each sentence in the training text; step R2, inputting a plurality of training natural language labels into the pre-training natural language model B to obtain feature representations corresponding to each training natural language label as a feature representation set; step R3, expanding each word in the sentence to the dimension of the size of the feature representation set, and performing dot product calculation with the feature representation set, and marking the word according to the training natural language label corresponding to the maximum value obtained by calculation; and step R4, calculating the loss according to the natural language label of the word and the corresponding actual label, and updating the parameters of the pre-training natural language model A and the pre-training natural language model B according to back propagation to obtain the pre-training natural language model A and the pre-training natural language model B after training.

[0016] In the vertical field knowledge graph construction method based on migration learning provided by the application, the marking method of the word according to the natural language label can be BIO marking method, and two-dimensional labels are marked on the word, one is the type of the entity, and the other is whether the word belongs to the beginning or middle of the type.

[0017] Effects of the application

[0018] According to the vertical field knowledge graph construction method based on transfer learning related to the present application, on the one hand, the pre-trained natural language model in the entity extraction is more suitable for the characteristics of the long-tail data in the vertical field by transferring the pre-trained natural language model in the general field to the vertical field through transfer learning and training the pre-trained natural language model through a small sample, and on the other hand, the relationship extraction is performed on any sentence through the combination of the Prompt template and the pre-trained natural language model and then the Zero-Shot inference, so that the model does not need to be retrained only by expanding the category dictionary, and the scalability of the relationship extraction is improved. Therefore, the vertical field entity extraction based on the transfer learning related to the present application has good efficiency and quality, and has good scalability in the relationship extraction. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a process schematic diagram of the pre-trained natural language models A, B, C and D obtained by transferring the BERT model in the embodiments of the present application;

[0020] Figure 2 is a schematic diagram of transferring the BERT model according to the training data set in the embodiments of the present application;

[0021] Figure 3 is a process schematic diagram of the entity extraction training of the pre-trained natural language model A and the pre-trained natural language model B in the embodiments of the present application;

[0022] Figure 4 is a process schematic diagram of the vertical field knowledge graph construction method based on transfer learning in the embodiments of the present application;

[0023] Figure 5 is a schematic diagram of the natural language label marking token process in the embodiments of the present application;

[0024] Figure 6 is a schematic diagram of the process of obtaining the relationship classification result between the tokens in the sentence in the embodiments of the present application. DETAILED DESCRIPTION

[0025] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the following embodiments combined with the drawings will specifically describe the vertical field knowledge graph construction method based on transfer learning of the present application.

[0026] The Linux server with GPU3090 is used as the hardware environment in the embodiment, the vertical field is the medical field, the knowledge text and the knowledge text in the training data set and the entity training data set are various disease-related medical data of disease encyclopedia websites such as Good Doctor Online and 39 Health Network, and part of experimental data provided by the project cooperation party including medical disease diagnosis and treatment specifications, the natural language label is obtained by processing the label information in the knowledge text and the database of knowledge extraction platforms such as Pytorch, for example: the label information "B-bacteria" is processed and converted into the natural language label "start position bacteria", and the existing relationship word is the relationship word extracted from the knowledge text.

[0027] Figure 1 is a process schematic diagram of the pre-training natural language model A, B, C and D obtained by performing transfer learning on the BERT model in the embodiment of the application.

[0028] Figure 2 is a schematic diagram of performing transfer learning on the BERT model according to the training data set in the embodiment of the application.

[0029] As shown in Figure 1 and Figure 2 , the training data set includes the training data Text of the vertical field and the enhanced training data Text' obtained by processing the training data Text according to the enhancement algorithm, and the process of obtaining the pre-training natural language model A, the pre-training natural language model B, the pre-training natural language model C and the pre-training natural language model D by performing transfer learning on the BERT model according to the training data set includes the following steps:

[0030] Step T1, inputting the training data Text into the BERT model L and regularizing the output result to obtain the sentence feature representation , inputting the enhanced training data Text' into the BERT model T and regularizing the output result to obtain the sentence feature representation . Each sentence feature representation of the training data Text is a positive sample only with the corresponding sentence feature representation in the enhanced training data Text', and is a negative sample with other sentence feature representations, for example, the sentence feature representation is a positive sample with the sentence feature representation , and the sentence feature representation is a negative sample with the sentence feature representation .

[0031] Step T2, obtaining an n*n matrix according to the sentence feature representation and the sentence feature representation , in which the diagonal line The sentence feature representations of other positions are negative samples, and the parameters of the BERT model L and the BERT model T are updated respectively through similarity calculation and loss calculation, to obtain the BERT model L and the BERT model T after transfer learning.

[0032] The pre-training natural language model A is any one of the BERT model L and the BERT model T after transfer learning, the pre-training natural language model B is the other one of the BERT model L and the BERT model T after transfer learning, the pre-training natural language model C is the BERT model T after transfer learning, and the pre-training natural language model D is the BERT model L after transfer learning.

[0033] The BERT model, the BERT model L and the BERT model T are all pre-training natural language models RoBERTa-wwm-ext.

[0034] The enhanced algorithm is back translation.

[0035] Figure 3 is a flowchart of entity extraction training of the pre-training natural language model A and the pre-training natural language model B in the embodiment of the application.

[0036] As shown in Figure 3 After obtaining the pre-training natural language model A and the pre-training natural language model B through transfer learning, the pre-training natural language model A and the pre-training natural language model B are trained according to the entity training data set, the entity training data set includes training text, training natural language labels and actual labels of each word in the training text, and includes the following steps:

[0037] Step R1, input the training text into the pre-training natural language model A to obtain each word in each sentence in the training text.

[0038] Step R2, input a plurality of training natural language labels into the pre-training natural language model B to obtain feature representations corresponding to each training natural language label as a feature representation set.

[0039] Step R3, expand each word in the sentence to the dimension of the size of the feature representation set, and perform dot product calculation with the feature representation set, and mark the word according to the training natural language label corresponding to the maximum value obtained by calculation.

[0040] The marking method of the natural language label for marking the word is BIO marking method, and two-dimensional labels are marked on the word, one is the type of the entity, and the other is whether the word belongs to the beginning or middle of the type.

[0041] Step R4, according to the natural language label of the word piece and the corresponding actual label, calculating the loss and updating the parameters of the pre-trained natural language model A and the pre-trained natural language model B according to the back propagation, obtaining the trained pre-trained natural language model A and the pre-trained natural language model B.

[0042] Figure 4 is a flowchart of the method for constructing a vertical field knowledge graph based on transfer learning in the embodiment of the application.

[0043] As Figure 4 shown, the method for constructing a vertical field knowledge graph based on transfer learning in the embodiment of the application is used to construct a vertical field knowledge graph according to a predetermined vertical field knowledge text, a natural language label representing an entity category and an existing relationship word, and includes the following steps:

[0044] Step S1, inputting the knowledge text into the pre-trained natural language model A to obtain each word piece of each sentence in the knowledge text.

[0045] Step S2, inputting a plurality of natural language labels into the pre-trained natural language model B to obtain feature representations corresponding to each natural language label as a feature representation set.

[0046] Step S3, expanding the word piece into a dimension of the size of the feature representation set and performing a dot product calculation with the feature representation set, marking the word piece with a natural language label corresponding to the feature representation according to the maximum weight obtained by the calculation, as a classification result of the word piece.

[0047] Among them, the marking method of marking the word piece according to the natural language label is the BIO marking method, which marks the word piece with two-dimensional labels, one is the type of entity, and the other is whether the word piece belongs to the beginning or middle of the type.

[0048] The way of expanding the word piece into a dimension of the size of the feature representation set is as follows: the output of the pre-trained natural language model A is a three-dimensional tensor, which is batch size, sequence length and hidden layer dimension respectively, and the hidden layer dimension of the pre-trained natural language model A in the embodiment is 768. By connecting a fully connected layer after the output layer of the pre-trained natural language model A, the dimension of each word piece is matched with the dimension of the label set.

[0049] Figure 5 is a schematic diagram of the process of marking the word piece according to the natural language label in the embodiment of the application.

[0050] As Figure 5As shown, the knowledge text "[CLS] The main pathogens of pneumonia include respiratory viruses, etc. [SEP]" is input into the pre-trained natural language model A, which obtains each word unit of the knowledge text, namely "lung", "inflammation", "of", ..., "etc." Multiple natural language labels, namely "starting position bacteria", "intermediate position bacteria", "starting position virus", "intermediate position virus", etc., are input into the pre-trained natural language model B, which obtains the feature representations 1, 2, 3, 4, 5, etc., corresponding to each natural language label as a feature representation set. The classification result is obtained by calculating the dot product. For example, the two-dimensional labels for the word unit "lung" are "disease class" and "starting position", and the two-dimensional labels for the word unit "inflammation" are "disease class" and "intermediate position". Both the pre-trained natural language model A and the pre-trained natural language model B are pre-trained natural language models RoBERTa-wwm-ext.

[0051] Step S4: Fill the m existing relation words into the Prompt template to obtain m relation sentences. Input the m relation sentences into the pre-trained natural language model C to obtain the vector representation of the m relation sentences.

[0052] Step S5: Input the sentence after the word tag classification result into the pre-trained natural language model D for encoding to obtain the encoding result.

[0053] Step S6: Calculate the similarity between the encoding result and the vector representations of the m relational sentences, and select the existing relational word corresponding to the maximum similarity calculation result as the relational classification result between the words in the sentence.

[0054] Figure 6 This is a schematic diagram illustrating the process of obtaining the classification results of the relationships between word elements in a sentence in an embodiment of the present invention.

[0055] like Figure 6 As shown, n existing relational terms Fill in the Prompt template in sequence: "{R} are two entities in the sentence" <e1> <e2>a relationship between the two, R represents an existing relationship word, <e1>and <e2>are two entities corresponding to R, i.e., the classification result, via a pre-trained natural language model C, to obtain a vector representation of the corresponding n relationship sentences the sentence after the token label classification result <e1>Breast milk jaundice <e1>Improper handling can cause complications such as: <e2>Intellectual, auditory, and nervous system impairment <e2>"The input is encoded into the pre-trained natural language model D to obtain the encoding result." The decoding result Respectively with vector representation Perform similarity calculation, and analyze the results. Select the maximum value The results showed that "complications" was used as the classification result of the relationship between "breast milk jaundice" and "intelligence, hearing and nervous system damage". Both the pre-trained natural language model C and the pre-trained natural language model D were pre-trained natural language models RoBERTa-wwm-ext.

[0056] Step S7: Construct tuples based on the word elements and corresponding relation classification results in the sentence, which serve as the knowledge extraction results of the sentence. Construct a knowledge graph for the vertical domain using the knowledge extraction results of all sentences in the knowledge text.

[0057] In this embodiment, various disease-related medical data from disease encyclopedia websites such as Haodf.com and 39.net, as well as some experimental data provided by project partners, including medical disease diagnosis and treatment guidelines, were used as test data. The pre-trained natural language model RoBERTa-wwm-ext of this invention, along with the Bi-LSTM-CRF model, BERT-CRF model, and BERT-base model after transfer learning, were used to label words with natural language tags. Precision (P), recall (R), and F1 score were used as evaluation criteria for the labeling results. The precision (P), recall (R), and F1 scores of the labeling results are as follows:

[0058]

[0059] The first column lists the names of each model, and the second to fourth columns show the precision (P), recall (R), and F1 score of the corresponding models, respectively. For example, the cell in the fifth row and second column shows the precision (P) of the pre-trained natural language model RoBERTa-wwm-ext used in this invention for natural language tagging of lexical units, which is 79.10. As can be seen from the table, the precision (P), recall (R), and F1 score of the pre-trained natural language model RoBERTa-wwm-ext used in this invention are all higher than the other three models, indicating that the pre-trained natural language model RoBERTa-wwm-ext performs better than the other three models in natural language tagging of lexical units.

[0060] The pre-training natural language model RoBERTa-wwm-ext of the present application is trained in small sample entity extraction with BERT-base model, Prototypical Network model and TANL model, the sample number k-shot of training is respectively 1 Shot, 10 Shot, 20 Shot, 50 Shot and Full Dataset, that is, all data in the test data as samples, the token of natural language label marking of each model after entity extraction training under different sample numbers is carried out, and the F1 value of the marking result is as follows:

[0061]

[0062] The first column is the name of each model, and the second to sixth columns correspond to the F1 value of the model under different training sample numbers. For example, the cell in the second column of the fifth row indicates that the pre-training natural language model RoBERTa-wwm-ext of the present application is trained in entity extraction under 1 Shot sample, and the F1 value of the token of natural language label marking result after training is 35.6. From the table, it can be seen that the pre-training natural language model RoBERTa-wwm-ext adopted by the present application has better effect on the token of natural language label marking result compared with the other three models when the sample of entity training is small sample.

[0063] On the basis of obtaining the relationship classification result between tokens, the pre-training natural language model RoBERTa-wwm-ext of the present application combined with Prompt template, that is, RoBERTa-Prompt, and the Bi-LSTM model after transfer learning, BERT-base model, RoBERTa model and BERT-Prompt model are used to calculate the relationship classification result between tokens in a sentence when the number of existing relationship words m is 5, 10, 15 and 20, and the precision P, recall R and F1 value of the classification result when the number of existing relationship words m is 5 and 10 are as follows:

[0064]

[0065] The first column is the name of each model, and the second to fourth columns are respectively the precision P, recall R and F1 value of the relationship classification result of the corresponding model when m=5, and the fifth to seventh columns are respectively the precision P, recall R and F1 value of the relationship classification result of the corresponding model when m=10. For example, the cell in the second column of the seventh row is the precision P of the relationship classification result of the RoBERTa-Prompt model of the present application when m=5.

[0066] The precision P, recall R and F1 value of the classification results of the existing relationship words with the number m of 15 and 20 are as follows:

[0067]

[0068] The first column is the name of each model, the second to fourth columns are the precision P, recall R and F1 value of the relationship classification results of the corresponding model when m = 15, and the fifth to seventh columns are the precision P, recall R and F1 value of the relationship classification results of the corresponding model when m = 20. For example, the cell in the second column of the seventh row is the precision P of the relationship classification results of the RoBERTa-Prompt model of the application when m = 15, which is 55.54. It can be seen from the table that the RoBERTa-Prompt model of the application has better relationship classification results than the other four models under different existing relationship word quantities.

[0069] Effects of the embodiments

[0070] According to the vertical field knowledge graph construction method based on transfer learning related to the embodiment, on the one hand, the pre-trained natural language model in the general field is transferred to the vertical field through transfer learning, and the pre-trained natural language model is trained through small samples, so that the pre-trained natural language model in entity extraction is more suitable for the characteristics of the long tail of the vertical field data, and on the other hand, the Prompt template is combined with the pre-trained natural language model, and then the Zero-Shot inference is used to extract the relationship of any sentence, so that only the category dictionary needs to be expanded, and the model does not need to be retrained, thereby improving the scalability of relationship extraction. In summary, the method has good efficiency and quality for entity extraction in the vertical field, and has good scalability in relationship extraction.

[0071] The above embodiments are preferred cases of the application and do not limit the protection scope of the application. < / e1>

Claims

1. A method for constructing a vertical domain knowledge graph based on transfer learning, used to construct a knowledge graph of the vertical domain based on predetermined vertical domain knowledge text, natural language labels representing entity categories, and existing relational terms, characterized in that, Includes the following steps: Step S1: Input the knowledge text into the pre-trained natural language model A to obtain each word unit of each sentence in the knowledge text; Step S2: Input the multiple natural language labels into the pre-trained natural language model B to obtain the feature representations of each natural language label as a feature representation set; Step S3: Expand the word element to the dimension of the size of the feature representation set, and perform a dot product calculation with the feature representation set. Label the word element with the natural language label corresponding to the feature representation with the maximum weight obtained by calculation, and use it as the classification result of the word element. Step S4: Fill the m existing relation words into the Prompt template to obtain m relation sentences, and input the m relation sentences into the pre-trained natural language model C to obtain the vector representation of the m relation sentences; Step S5: Input the sentence after classifying the lexical tags into the pre-trained natural language model D for encoding to obtain the encoding result; Step S6: Calculate the similarity between the encoding result and the vector representation of the m relational sentences respectively, and select the existing relational word corresponding to the maximum value of the similarity calculation result as the relational classification result between the word elements in the sentence; Step S7: Construct tuples based on the lexical units and corresponding relation classification results in the sentence, as the knowledge extraction result of the sentence. Then, construct the knowledge graph for the vertical domain using the knowledge extraction results of all sentences in the knowledge text. The pre-trained natural language model A, pre-trained natural language model B, pre-trained natural language model C, and pre-trained natural language model D are obtained from the training dataset through transfer learning of the BERT model. The training dataset includes training data for the vertical domain and augmented training data processed by the augmentation algorithm. The process of obtaining the pre-trained natural language model A, the pre-trained natural language model B, the pre-trained natural language model C, and the pre-trained natural language model D by performing transfer learning on the BERT model based on the training dataset includes the following steps: Step T1: Input the training data into the BERT model L and regularize the output to obtain the sentence feature representation. The enhanced training data is input into the BERT model T and the output is regularized to obtain the sentence feature representation. ; Step T2, based on the sentence feature representation and the sentence feature representation By calculating similarity and loss, the parameters of the BERT model L and the BERT model T are updated respectively, resulting in the BERT model L and the BERT model T after transfer learning. If the pre-trained natural language model A is either the BERT model L or the BERT model T after transfer learning, then the pre-trained natural language model B is the other one of the BERT model L or the BERT model T after transfer learning, the pre-trained natural language model C is the BERT model T after transfer learning, and the pre-trained natural language model D is the BERT model L after transfer learning.

2. The method for constructing a vertical domain knowledge graph based on transfer learning according to claim 1, characterized in that: in, The enhancement algorithm is a back-translation method.

3. The method for constructing a vertical domain knowledge graph based on transfer learning according to claim 1, characterized in that: in, The BERT model, BERT model L, and BERT model T are all pre-trained natural language models RoBERTa-wwm-ext.

4. The method for constructing a vertical domain knowledge graph based on transfer learning according to claim 1, Its features are: After obtaining the pre-trained natural language model A and the pre-trained natural language model B through transfer learning, the pre-trained natural language model A and the pre-trained natural language model B are trained based on an entity training dataset. The entity training dataset includes training text, training natural language labels, and the actual labels of each word in the training text, and includes the following steps: Step R1: Input the training text into the pre-trained natural language model A to obtain each word unit of each sentence in the training text; Step R2: Input the multiple training natural language labels into the pre-trained natural language model B to obtain the feature representations corresponding to each training natural language label as a feature representation set; Step R3: Expand each word element in the sentence to the dimension of the size of the feature representation set, and perform a dot product calculation with the feature representation set. Label the word element with the training natural language label corresponding to the maximum value obtained from the calculation. Step R4: Calculate the loss based on the natural language label and the corresponding actual label of the word, and update the parameters of the pre-trained natural language model A and the pre-trained natural language model B according to backpropagation, to obtain the trained pre-trained natural language model A and the pre-trained natural language model B.

5. The method for constructing a vertical domain knowledge graph based on transfer learning according to claim 1 or claim 4, characterized in that: in, The tagging method for the natural language tags is the BIO tagging method, which assigns two-dimensional tags to the lexicons: one is the type of entity, and the other is whether the lexicon belongs to the beginning or middle of the type.

Images