Knowledge graph construction method and device, equipment and storage medium

Abstract

Disclosed are a knowledge graph construction method, apparatus and device, and a storage medium, the method comprising: acquiring a first predicate set, the first predicate set comprising a plurality of predicates in a target domain, the predicates being predicates in a triple of a knowledge graph, the triple comprising a subject, a predicate and an object; based on the first predicate set, training a teacher large language model, the teacher large language model being used for extracting a plurality of triads from the target domain knowledge text, and predicates in any one of the triads extracted by the teacher large model being in the first predicate set; based on the teacher big language model, performing knowledge distillation on the student big language model; and based on the student large language model after knowledge distillation, constructing a target domain knowledge graph. According to the method, the knowledge graph can be constructed based on the large language model in a low-resource environment.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a knowledge graph construction method, device, equipment and storage medium. Background Art

[0002] A knowledge graph is a graph composed of multiple triples. A triple includes three parts: subject, predicate, and object. Subject and object are entities, and predicate is a relationship. Therefore, when constructing a knowledge graph, it is necessary to accurately obtain multiple triples to construct a complete knowledge graph.

[0003] In the related art, the knowledge graph construction method includes: obtaining multiple entities, using LLM (Large Language Model) to extract the relationship between two entities among the multiple entities, thereby obtaining multiple triples, and constructing a knowledge graph based on the multiple triples.

[0004] However, LLM is a large language model with large volume, which is difficult to use in low-resource environments. Summary of the invention

[0005] The present disclosure provides a knowledge graph construction method, apparatus, device and storage medium, which can realize the construction of a knowledge graph based on a large language model in a low-resource environment. The technical solution includes at least the following solutions: In a first aspect, a method for constructing a knowledge graph is provided, comprising: obtaining a first predicate set, the first predicate set including multiple predicates in a target domain, the predicates being predicates in triples of the knowledge graph, the triples including a subject, a predicate, and an object; based on the first predicate set, training a teacher big language model, the teacher big language model being used to extract multiple triples from a target domain knowledge text, and the predicates in any triple extracted by the teacher big model are all in the first predicate set; based on the teacher big language model, performing knowledge distillation on a student big language model; and constructing a target domain knowledge graph based on the student big language model after knowledge distillation.

[0006] Optionally, training the teacher large language model based on the first predicate set includes: obtaining a first fine-tuning instruction, the first fine-tuning instruction including a first target domain knowledge text and a label, the first fine-tuning instruction being used to control the teacher large language model to extract multiple triples from the first target domain knowledge text in a format indicated by the label, and the predicates in any triple extracted by the teacher large language model from the first target domain knowledge text are all in the first predicate set; based on the first fine-tuning instruction, fine-tuning the teacher large language model in a low-order adaptive manner.

[0007] Optionally, based on the first fine-tuning instruction, the teacher large language model is fine-tuned in a low-order adaptive manner, including: preprocessing the first fine-tuning instruction, the preprocessing including obtaining the word sequence, attention mask and label of the first fine-tuning instruction; inputting the word sequence, attention mask and label of the first fine-tuning instruction into the teacher large language model to fine-tune the teacher large language model.

[0008] Optionally, performing knowledge distillation on the student large language model based on the teacher large language model includes: The following formula is used as the loss function to perform knowledge distillation on the student large language model:

[0009] in, is the loss value of the loss function, is the probability distribution output by the teacher’s large language model, is the probability distribution output by the student language model, represents an inverse Kullback-Leibler divergence transformation of the output of the teacher large language model and the output of the student large language model, Indicated in For any sample in the training set, In the case of samples generated by the probability distribution output by the student large language model, and joint expectations.

[0010] Optionally, the method further includes: obtaining evaluation indicators of the student large language model, the evaluation indicators including accuracy, recall rate and F1 score; and evaluating the performance of the student large language model based on the evaluation indicators.

[0011] In the second aspect, a knowledge graph construction device is also provided, including: an acquisition module, used to acquire a first predicate set, the first predicate set including multiple predicates in the target domain, the predicates are predicates in triples of the knowledge graph, and the triples include subject, predicate, and object; a training module, used to train a teacher large language model based on the first predicate set, the teacher large language model is used to extract multiple triples from the target domain knowledge text, and the predicates in any triple extracted by the teacher large model are in the first predicate set; a knowledge distillation module, used to perform knowledge distillation on the student large language model based on the teacher large language model; a knowledge graph construction module, used to construct a target domain knowledge graph based on the student large language model after knowledge distillation.

[0012] Optionally, the training module is also used to obtain a first fine-tuning instruction, which includes a first target domain knowledge text and a label, and the first fine-tuning instruction is used to control the teacher large language model to extract multiple triples from the first target domain knowledge text in the format indicated by the label, and the predicates in any triples extracted by the teacher large language model from the first target domain knowledge text are in the first predicate set; based on the first fine-tuning instruction, the teacher large language model is fine-tuned in a low-order adaptive manner.

[0013] Optionally, the training module is also used to preprocess the first fine-tuning instruction, and the preprocessing includes obtaining the word sequence, attention mask and label of the first fine-tuning instruction; inputting the word sequence, attention mask and label of the first fine-tuning instruction into the teacher large language model to fine-tune the teacher large language model.

[0014] Optionally, the knowledge distillation module is further used to use the following formula as a loss function to perform knowledge distillation on the student large language model:

[0015] in, is the loss value of the loss function, is the probability distribution output by the teacher’s large language model, is the probability distribution output by the student language model, represents an inverse Kullback-Leibler divergence transformation of the output of the teacher large language model and the output of the student large language model, Indicated in For any sample in the training set, In the case of samples generated by the probability distribution output by the student large language model, and joint expectations.

[0016] Optionally, the device also includes: an evaluation module, which is used to obtain evaluation indicators of the student large language model, wherein the evaluation indicators include accuracy, recall rate and F1 score; based on the evaluation indicators, the performance of the student large language model is evaluated.

[0017] In a third aspect, a computer device is also provided, including: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor, thereby executing the knowledge graph construction method described in the above embodiment.

[0018] In a fourth aspect, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor to execute the knowledge graph construction method described in the above embodiment.

[0019] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0020] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least: In the disclosed embodiment, the teacher's large language model is trained by the first predicate set, so that the trained teacher's large language model can more accurately extract multiple triples required for the target domain knowledge graph from the target domain knowledge text; and the knowledge of the teacher's large language model can be transferred to the student's large language model by performing knowledge distillation on the student's large language model. The student's large language model is usually a small-volume large language model, and a small-volume large language model can run in a low-resource environment, so the student's large language model after knowledge distillation can be used to build a target domain knowledge graph in a low-resource environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a knowledge graph construction method provided by an exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart of a knowledge graph construction method provided by another exemplary embodiment of the present disclosure is shown; Figure 3 is a schematic diagram of the first fine-tuning instruction; Figure 4 A schematic diagram of the structure of a knowledge graph construction device provided by an exemplary embodiment of the present disclosure is shown; Figure 5 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Unless otherwise defined, the technical terms or scientific terms used herein shall have the usual meanings understood by persons with ordinary skills in the field to which the present disclosure belongs. The words "first", "second", "third" and similar words used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate a quantity limitation, but indicate the existence of at least one. Words such as "include" or "comprise" and similar words mean that the elements or objects appearing before "include" or "comprise" include the elements or objects listed after "include" or "comprise" and their equivalents, and do not exclude other elements or objects. Words such as "connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. A and / or B means that there are three situations: A, B, and A and B.

[0024] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A flowchart of a knowledge graph construction method provided by an exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 1 , the method comprising: In step 101, a first predicate set is obtained.

[0026] The first predicate set includes multiple predicates in the target domain, and the predicates are predicates in triples of the knowledge graph, and the triples include a subject, a predicate, and an object.

[0027] A triple in the knowledge graph is two entities and the relationship between them. The subject and the object are the two entities, and the predicate is the relationship between them.

[0028] In the disclosed embodiment, the target domain is the domain to which the constructed knowledge graph belongs. For example, if a knowledge graph in the field of an industrial chain needs to be constructed, the target domain is the field of the industrial chain.

[0029] The following explanation is given using the target field as the industrial chain field. If the target field is other fields, the processing method of the industrial chain field can also be referred to.

[0030] Table 1 is the first predicate set in the field of industrial chain.

[0031]

[0032] Table 1 includes a total of 29 predicates in the industrial chain field, which are extracted from the REBEL (Relation Extraction By End-to-end Language generation) dataset. There are many related technologies on how to obtain the REBEL dataset, so we will not elaborate on it here.

[0033] When obtaining the first predicate set, the number of times each predicate in other knowledge graphs in the target field is used can be counted, and the predicates whose number of uses is greater than the first number threshold are stored in the first predicate set. The number of the first number threshold can be set according to experience, and the embodiment of the present disclosure does not limit this. In addition, the user can also add or delete the predicates in the first predicate set as needed.

[0034] In step 102, a teacher language model is trained based on the first predicate set.

[0035] The teacher's large language model is used to extract multiple triples from the target domain knowledge text, and the predicates in any triple extracted by the teacher's large model are in the first predicate set.

[0036] In step 103, knowledge distillation is performed on the student large language model based on the teacher large language model.

[0037] Knowledge distillation is a machine learning model compression method used to transfer the knowledge of a large model (the teacher large language model) to a smaller model (the student large language model).

[0038] In step 104, a target domain knowledge graph is constructed based on the student language model after knowledge distillation.

[0039] In the disclosed embodiment, the teacher's large language model is trained by the first predicate set, so that the trained teacher's large language model can more accurately extract multiple triples required for the target domain knowledge graph from the target domain knowledge text; and the knowledge of the teacher's large language model can be transferred to the student's large language model by performing knowledge distillation on the student's large language model. The student's large language model is usually a small-volume large language model, and a small-volume large language model can run in a low-resource environment, so the student's large language model after knowledge distillation can be used to build a target domain knowledge graph in a low-resource environment.

[0040] Figure 2 A flowchart of a knowledge graph construction method provided by another exemplary embodiment of the present disclosure is shown, and the method can be executed by a computer device. Figure 2 , the method comprising: In step 201, a first predicate set is obtained.

[0041] The relevant contents of step 201 refer to the aforementioned step 101, and the detailed description is omitted here.

[0042] In step 202, a teacher language model is trained based on the first predicate set.

[0043] In the disclosed embodiment, the teacher large language model is LLaMa-2-7B (Large Language Model MetaAI-2 -7B, the 2nd-7B generation large language model produced by Meta AI), and the number of parameters of the model is approximately 7 billion.

[0044] Optionally, step 202 includes the following steps ab: Step a, obtaining a first fine-tuning instruction.

[0045] The first fine-tuning instruction includes a first target domain knowledge text and a label. The first fine-tuning instruction is used to control the teacher large language model so that the teacher large language model extracts multiple triples from the first target domain knowledge text in the format indicated by the label, and the first fine-tuning instruction is also used to control the teacher large language model so that the predicates in any triple extracted by the teacher large language model from the first target domain knowledge text are in the first predicate set.

[0046] Optionally, when training the teacher's large language model, multiple knowledge texts in the second target field can be obtained, and the multiple knowledge texts in the second target field can be divided into a training set and a test set. The second target field knowledge text can be a natural language text related to the target field. For example, when the target field is the industrial chain, the second target field knowledge text can be an annual report of a manufacturing enterprise. The second target field knowledge text is usually long and has a large amount of content. In actual processing, the second target field knowledge text is often divided into multiple parts, and then these multiple parts are processed separately.

[0047] In this case, the first target domain knowledge text may be any one of the multiple parts of the multiple second target domain knowledge texts in the training set. Each of the multiple parts of the multiple second target domain knowledge texts in the training set corresponds to a fine-tuning instruction, so there are multiple fine-tuning instructions, and the first fine-tuning instruction is the fine-tuning instruction corresponding to the first target domain knowledge text among the multiple fine-tuning instructions.

[0048] For example, there are M second target domain knowledge texts in the training set, and each second target domain knowledge text is divided into N parts, so there are a total of M times N parts, each part corresponds to a fine-tuning instruction, so there are a total of M times N fine-tuning instructions.

[0049] Figure 3 This is a schematic diagram of the first fine-tuning instruction. Figure 3 It can be seen that the first fine-tuning instruction includes three parts: instruction, input and response.

[0050] The text in the instruction part is the action that the teacher model needs to perform. "Extract triplets from input text and output some lists like [ subject , predicate , object ] inorder they appear in the text, Predicates must be in the following list :{relations}." This sentence means that the teacher model needs to extract multiple triplets from the input first target domain knowledge text in the order in which the triplets appear in the first target domain knowledge text. The format of the triples is [subject, predicate, object], and the predicate in the triple must be in the list {relations}. Here, [subject, predicate, object] is the label, and the list {relations} is the first predicate set.

[0051] The input part is used to fill in the first target domain knowledge text, and the response part is used to fill in the response of the teacher's large language model (such as the triples extracted by the teacher's large language model).

[0052] Step b: Based on the first fine-tuning instruction, fine-tune the teacher's large language model using a low-order adaptive method.

[0053] Optionally, step b includes the following two steps: The first step is to preprocess the first fine-tuning instruction.

[0054] Preprocessing includes obtaining word sequences, attention masks, and labels for the first fine-tuning instruction.

[0055] Preprocessing the first fine-tuning instruction can be implemented by a tokenizer, wherein a tokenizer is a class for segmenting a text into multiple independent words or tags, i.e., tokens, and then converting multiple tokens into sequences. Here, converting multiple tokens into sequences means converting each token into an integer sequence in the vocabulary through a vocabulary.

[0056] There are many methods for implementing tokenizers and vocabularies in the related art, so detailed description is omitted here.

[0057] Through the tokenizer, the first fine-tuning instruction can be converted into input_ids (that is, a word sequence), attention_mask (that is, an attention mask), and labels (that is, labels). Input_ids is the integer sequence of each word in the dictionary in the first fine-tuning instruction. Attention_mask indicates which tokens participate in the operation and which are ignored in the attention mechanism operation of the teacher's large language model; usually, the token corresponding to the position of 1 will participate in the operation, while the token corresponding to the position of 0 will be ignored.

[0058] Labels is the result of replacing some positions of the instruction part of input_ids with -100. The part replaced with -100 does not calculate the loss in the autoregression process. Here, in the instruction part of input_ids, the label does not need to be replaced with -100, that is, the label needs to be generated, and the part other than the label in the instruction part needs to be replaced with -100, which does not need to be generated by the model autoregression.

[0059] In the second step, the word sequence, attention mask and label of the first fine-tuning instruction are input into the teacher large language model to fine-tune the teacher large language model.

[0060] Optionally, when fine-tuning the teacher large language model, LoRA (Low-Rank Adaptation of Large Language Models) is used to quantize the teacher large language model. When LoRA is used to quantize the teacher large language model, it is necessary to select appropriate hyperparameters and LoRA quantization hyperparameters.

[0061] For example, in the hyperparameters, the number of iterations (epoch) = 20, the batch size (batch_size) = 1, and the learning rate (learning_rate) = , gradient accumulation steps (gradient_accumulation_steps=2). In the LoRA quantization hyperparameters, r=4, lora_alpha=16, lora_dropout=0.15.

[0062] There are many related technologies about hyperparameters and how to select LoRA quantization hyperparameters, so detailed description is omitted here.

[0063] After the hyperparameters and LoRA quantization hyperparameters are set, the word sequence, attention mask and label of the first fine-tuning instruction can be input into the teacher large language model to fine-tune the teacher large language model.

[0064] For the fine-tuning instructions other than the first fine-tuning instruction among the multiple fine-tuning instructions, the above steps ab can also be adopted to achieve fine-tuning of the teacher's large language model.

[0065] In step 203, knowledge distillation is performed on the student large language model based on the teacher large language model.

[0066] In the disclosed embodiment, the student language model is a BART (Bidirectional and Auto-Regressive Transformers) model. The parameters of the BART model are in the tens of millions, about 17.5 million. Tens of millions is much smaller than 7 billion, so it can be seen that the parameters of the student language model are much smaller than those of the teacher language model, so the student language model is easy to adapt to low-resource environments.

[0067] In the disclosed embodiment, the student language model learns to match the probability distribution of the teacher language model output by the knowledge distillation of soft labels. Here, the soft label is the output result of the teacher language model as a probability. For example, the probability that the predicate between a subject and an object is A is 0.9, which is a soft label.

[0068] Optionally, when performing knowledge distillation, temperature adjustment (also known as warming) can also be performed to adjust the smoothness of the soft labels output by the teacher's large language model.

[0069] There are many ways to implement knowledge distillation and temperature adjustment of soft tags in related technologies, so detailed description is omitted here.

[0070] Optionally, in the disclosed embodiment, formula (1) is used as the loss function to perform knowledge distillation on the student large language model.

[0071] (1) In formula (1), is the loss value of the loss function, is the probability distribution output by the teacher’s large language model, is the probability distribution output by the student's large language model, Indicates the inverse Kullback-Leibler divergence transformation of the output of the teacher's large language model and the output of the student's large language model. Indicated in For any sample in the training set, In the case of samples generated by the probability distribution output by the student language model, and joint expectations.

[0072] Among them, the inverse KL divergence transformation is to perform an inverse operation transformation on the KL divergence transformation output, that is, Operation transformation, usually in the generative model, uses the inverse KL divergence transformation as the loss function. The generative model here is such as the teacher's large language model and the student's large language model in the embodiment of the present disclosure.

[0073] Through step 203, knowledge distillation can be performed on the student language model. After knowledge distillation is performed on the student language model, the performance of the student language model needs to be evaluated through steps 204 to 205. If the performance of the student language model is unqualified (for example, overfitting or underfitting), the parameters of the student language model training need to be adjusted and steps 203 to 205 need to be re-executed until the performance of the student language model is qualified. At this time, knowledge distillation is completed and step 206 can be executed.

[0074] In step 204, the evaluation index of the student large language model is obtained.

[0075] When obtaining the evaluation index, multiple second target domain knowledge texts in the test set are input into the student large language model, and the performance of the student large language model is evaluated based on the output of the student large language model.

[0076] Optionally, the evaluation index includes accuracy, recall rate and F1 score. The evaluation index can be calculated by formula (2).

[0077] (2) In formula (2), For accuracy, (True Positive) is the number of positive examples correctly predicted by the student language model; (False Positive), which is the number of positive examples incorrectly predicted by the student language model.

[0078] The prediction results output by the student language model include positive and negative classes. If the prediction result of a triplet output by the student language model is positive, and the triplet is actually positive, then the prediction result belongs to , if the prediction result of a triplet output by the student language model is positive, but in fact the triplet is negative, then the prediction result belongs to .

[0079] The recall rate can be calculated using formula (3).

[0080] (3) In formula (3), is the recall rate, (True Negative, true negative examples), that is, the number of positive examples correctly predicted by the student language model. The meanings of other parameters in formula (3) are the same as those in formula (2), and detailed description is omitted here.

[0081] If the prediction result of a triplet output by the student language model is negative, and the triplet is actually negative, then the prediction result belongs to .

[0082] The F1 score can be calculated using formula (4).

[0083] (4) In formula (4), is the F1 score. The meanings of other parameters in formula (4) are the same as those in formula (2) and formula (3), and their detailed description is omitted here.

[0084] In step 205, the performance of the student large language model is evaluated based on the evaluation indicators.

[0085] There are many related technologies on how to evaluate the performance of the student large language model based on evaluation indicators, so detailed description is omitted here.

[0086] If the performance of the student language model is unsatisfactory, the parameters of the student language model training, such as learning rate, number of training epochs, weight decay, batch size, etc., can be adjusted, and then steps 203 to 205 can be re-executed.

[0087] After the performance of the student large language model is qualified, step 206 can be executed.

[0088] In step 206, a target domain knowledge graph is constructed based on the student language model after knowledge distillation.

[0089] The student large language model after knowledge distillation can better inherit the performance of the teacher large language model. Since the parameter scale of the student large language model is small, the student large language model after knowledge distillation can be deployed in a low-resource or no-network environment to accurately extract triples from the target domain knowledge text.

[0090] During implementation, for the student language model, the target domain knowledge text can also be input into the student language model in the form of fine-tuning instructions.

[0091] In addition, the output of the student large language model is in encoded form, and the output of the student large language model needs to be decoded; after decoding, multiple triplets can be obtained, and these multiple triplets can be visualized through the graph database, so as to build a knowledge graph in the target domain.

[0092] The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

[0093] Figure 4 A schematic diagram of a knowledge graph construction device provided by an exemplary embodiment of the present disclosure is shown. Figure 4 The knowledge graph construction device 400 includes: an acquisition module 401, a training module 402, a knowledge distillation module 403 and a knowledge graph construction module 404.

[0094] The acquisition module 401 is used to acquire a first predicate set, which includes multiple predicates in the target field. The predicates are predicates in triples of the knowledge graph, and the triples include a subject, a predicate, and an object.

[0095] The training module 402 is used to train the teacher's large language model based on the first predicate set. The teacher's large language model is used to extract multiple triples from the target domain knowledge text, and the predicates in any triple extracted by the teacher's large model are in the first predicate set.

[0096] The knowledge distillation module 403 is used to perform knowledge distillation on the student large language model based on the teacher large language model.

[0097] The knowledge graph construction module 404 is used to construct a target domain knowledge graph based on the student language model after knowledge distillation.

[0098] Optionally, the training module 402 is also used to obtain a first fine-tuning instruction, which includes a first target domain knowledge text and a label. The first fine-tuning instruction is used to control the teacher's large language model to extract multiple triples from the first target domain knowledge text in the format indicated by the label, and the predicates in any triple extracted by the teacher's large language model from the first target domain knowledge text are in the first predicate set; based on the first fine-tuning instruction, the teacher's large language model is fine-tuned in a low-order adaptive manner.

[0099] Optionally, the training module 402 is also used to preprocess the first fine-tuning instruction, and the preprocessing includes obtaining the word sequence, attention mask and label of the first fine-tuning instruction; inputting the word sequence, attention mask and label of the first fine-tuning instruction into the teacher large language model to fine-tune the teacher large language model.

[0100] Optionally, the knowledge distillation module 403 is further configured to use the following formula as a loss function to perform knowledge distillation on the student large language model:

[0101] in, is the loss value of the loss function, is the probability distribution output by the teacher’s large language model, is the probability distribution output by the student's large language model, Indicates the inverse Kullback-Leibler divergence transformation of the output of the teacher's large language model and the output of the student's large language model. Indicated in For any sample in the training set, In the case of samples generated by the probability distribution output by the student language model, and joint expectations.

[0102] Optionally, the device further includes: an evaluation module 405 .

[0103] The evaluation module 405 is used to obtain evaluation indicators of the student large language model, the evaluation indicators include accuracy, recall rate and F1 score; based on the evaluation indicators, the performance of the student large language model is evaluated.

[0104] It should be noted that: when the knowledge graph construction device provided in the above embodiment constructs a knowledge graph, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the knowledge graph construction device provided in the above embodiment and the knowledge graph construction method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0105] The division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present disclosure may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0106] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, mobile phone, or communication device, etc.) or a processor (processor) to execute all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0107] Figure 5 Schematic diagram of the structure of the computer device provided by the embodiment of the present disclosure. Figure 5 As shown, the computer device 500 includes: a processor 501 and a memory 502 .

[0108] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0109] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement the knowledge graph construction method provided in the embodiments of the present disclosure.

[0110] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the computer device 500, and the computer device 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0111] The embodiments of the present disclosure also provide a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a computer device, the computer device can execute the knowledge graph construction method provided in the embodiments of the present disclosure.

[0112] The embodiments of the present disclosure also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the knowledge graph construction method provided in the embodiments of the present disclosure.

[0113] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A knowledge graph construction method, characterized in that: The method comprises: Acquire a first predicate set, wherein the first predicate set includes a plurality of predicates in the target domain, wherein the predicates are predicates in a triple of a knowledge graph, and the triple includes a subject, a predicate, and an object; Based on the first predicate set, a teacher language model is trained, where the teacher language model is used to extract multiple triples from the target domain knowledge text, and the predicates in any triple extracted by the teacher language model are all in the first predicate set; Based on the teacher's large language model, knowledge distillation is performed on the student's large language model; Based on the student language model after knowledge distillation, a knowledge graph of the target domain is constructed.

2. The method according to claim 1, characterized in that The step of training the teacher large language model based on the first predicate set includes: Obtaining a first fine-tuning instruction, wherein the first fine-tuning instruction includes a first target domain knowledge text and a label, and the first fine-tuning instruction is used to control the teacher large language model to extract a plurality of triples from the first target domain knowledge text according to the format indicated by the label, and the predicates in any triple extracted from the first target domain knowledge text by the teacher large language model are all in the first predicate set; Based on the first fine-tuning instruction, the teacher large language model is fine-tuned in a low-order adaptive manner.

3. The method according to claim 2, characterized in that The step of fine-tuning the teacher large language model based on the first fine-tuning instruction in a low-order adaptive manner includes: Preprocessing the first fine-tuning instruction, the preprocessing comprising obtaining a word sequence, an attention mask, and the label of the first fine-tuning instruction; The word sequence, attention mask and label of the first fine-tuning instruction are input into the teacher large language model to fine-tune the teacher large language model.

4. The method according to any one of claims 1 to 3, characterized in that: The performing knowledge distillation on the student large language model based on the teacher large language model includes: The following formula is used as the loss function to perform knowledge distillation on the student large language model: in, is the loss value of the loss function, is the probability distribution output by the teacher’s large language model, is the probability distribution output by the student language model, represents an inverse Kullback-Leibler divergence transformation of the output of the teacher large language model and the output of the student large language model, Indicated in For any sample in the training set, In the case of samples generated by the probability distribution output by the student large language model, and joint expectations.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Obtaining evaluation indicators of the student large language model, wherein the evaluation indicators include accuracy, recall rate and F1 score; Based on the evaluation index, the performance of the student large language model is evaluated.

6. A knowledge graph construction device, characterized in that: The device comprises: An acquisition module, configured to acquire a first predicate set, wherein the first predicate set includes a plurality of predicates in a target domain, wherein the predicates are predicates in a triple of a knowledge graph, wherein the triple includes a subject, a predicate, and an object; A training module, used for training a teacher language model based on the first predicate set, wherein the teacher language model is used for extracting a plurality of triples from the target domain knowledge text, and the predicates in any triple extracted by the teacher language model are all in the first predicate set; A knowledge distillation module, used for performing knowledge distillation on the student large language model based on the teacher large language model; The knowledge graph construction module is used to construct a target domain knowledge graph based on the student language model after knowledge distillation.

7. A computer device, characterized in that: The computer device comprises: a memory and a processor, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images