Processing method and device for relation extraction based on large language model

Abstract

The embodiment of the invention relates to a processing method and device for relation extraction based on a large language model. The method comprises the steps that the large language model is selected as a first model; constructing a first data set through a data acquisition / cleaning / labeling mode of a manual / machine model to pre-train a relationship extraction / verification function of the first model; constructing a second data set based on the text generation function of the first model after the pre-training is finished, performing data cleaning on the second data set based on the relationship verification function of the first model, and performing fine adjustment on the relationship extraction function of the first model based on the cleaned second data set; and after the fine tuning is finished, substituting the queried text input by the user, the limited relation sequence and the output format into the first instruction template to perform large-model input text assembly, and inputting the assembled text into the first model to perform relation triple recognition and screening processing to obtain an output text to be fed back to the user. According to the invention, the model generalization can be enhanced, and the prediction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a processing method and device for performing relationship extraction based on a large language model. Background Art

[0002] With the development of artificial intelligence technology, large language models (LLMs) have performed well in natural language processing (NLP) tasks. Large language models can understand and generate natural language text by learning massive amounts of text data, and are widely used in tasks such as text generation, translation, and question-answering systems. However, large language models still face some challenges in relation extraction (RE) tasks. The relation extraction task mentioned here refers to identifying the relationship between entities in the input text and outputting the corresponding recognition results, such as the relationship triple: head entity + entity relationship / association relationship + tail entity. Under normal circumstances, the training of the relation extraction function of a large language model is mostly based on supervised training, which requires a large amount of labeled data sets. At present, when preparing this labeled data set, most of them use manual data collection / labeling methods for processing. This conventional processing method is affected by human factors and is bound to have problems such as insufficient amount of collected data and insufficient collection fields. These problems will bring challenges to the model such as insufficient training, weak model generalization, poor training quality, and low model prediction accuracy. Summary of the invention

[0003] The purpose of the present invention is to provide a processing method, device, electronic device and computer-readable storage medium for relation extraction based on a large language model in view of the defects of the prior art. The present invention selects a type of large language model as the corresponding first model; and constructs a benchmark label data set as the corresponding first data set through data collection, data cleaning and data annotation of artificial or other machine models; and pre-trains the relation extraction function and relation verification function of the first model based on the first data set; and after the pre-training, constructs an enhanced data set based on the text generation function of the first model as the corresponding second data set; and cleans the second data set based on the relation verification function of the first model; and fine-tunes the relation extraction function of the first model based on the cleaned second data set; after the fine-tuning, the first query text, the first limited relation sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model for relation triple recognition and screening to obtain the corresponding first output text to feedback to the user. The present invention utilizes the text generation function of the large language model itself to add a method for constructing an enhanced data set on top of the processing mechanism of a conventional label data set, and improves the data accuracy of the enhanced data set by training the relationship verification function of the large language model. In this way, the enhanced data set with a large amount of data can be used to improve the training adequacy and training quality of the model, and achieve the purpose of enhancing the generalization of the model and improving the prediction accuracy of the model; in addition, the present invention also takes the relationship extraction function of the large language model as the core and provides users with a processing mechanism for identifying and screening relationship triples for specified query texts through a first instruction template, which can both simplify user operation steps and improve user experience.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention provides a processing method for relation extraction based on a large language model, the method comprising:

[0005] Select a class of large language models as the corresponding first model;

[0006] A benchmark label data set is constructed by data collection, data cleaning and data labeling by manual or other machine models, and recorded as the corresponding first data set; and a relationship extraction function of the first model is pre-trained based on the first data set; and after the relationship extraction function pre-training is completed, a relationship verification function of the first model is pre-trained based on the first data set;

[0007] After the pre-training of the relationship verification function is completed, an enhanced data set is constructed based on the text generation function of the first model and recorded as the corresponding second data set; and the second data set is cleaned based on the relationship verification function of the first model; and the relationship extraction function of the first model is fine-tuned based on the cleaned second data set;

[0008] After fine-tuning is completed, the first query text, the first restricted relationship sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text to be fed back to the user.

[0009] Preferably, the first model includes at least a GPT series model, a GPT-RE model, a BERT model, a UIE series model, a QA4RE model, and a Wenxin model;

[0010] The first data set includes a plurality of first data records; the first data record includes a first training text and a first label relationship triple sequence; the first label relationship triple sequence includes one or more first label relationship triples; the first label relationship triple consists of a head entity text, an entity relationship text and a tail entity text;

[0011] The second data set includes a plurality of second data records; the second data records include a second label relationship triple sequence and a second training text; the second label relationship triple sequence includes one or more second label relationship triples; the second label relationship triple consists of a head entity text, an entity relationship text and a tail entity text;

[0012] The first restricted relationship sequence consists of one or more first entity relationship texts;

[0013] The first output format at least includes an array sequence format with a head entity-entity relationship-tail entity field, a list format with a head entity-entity relationship-tail entity field, and a specified script format with a head entity-entity relationship-tail entity field; the specified script format at least includes a BAT script, an HTM script, a JSON script, and a Python script;

[0014] The first instruction template is a formatted text template, which is used to use the first query text, the first restricted relationship sequence and the first output format as the corresponding current query text, the current restricted relationship sequence and the current output format, and to inform the large language model through a preset narrative text to extract relation triples from the current query text, and to filter all extracted relation triples based on the current restricted relationship sequence, and to assemble and output the output text according to the current output format for the triple relationships that satisfy the current restricted relationship sequence.

[0015] Preferably, the reference label data set constructed by artificial or other machine model data collection, data cleaning and data labeling is recorded as the corresponding first data set, which specifically includes:

[0016] Performing big data collection of text information in multiple fields by manual or other machine model data collection methods to obtain corresponding multiple first collected texts;

[0017] and performing grammar and content compliance checks on all the first collected texts obtained by manual or other machine model data verification methods, and deleting the first collected texts that fail the grammar check or the content compliance check; and performing duplicate text elimination processing on the remaining first collected texts by manual or other machine model text deduplication methods; and performing text similarity calculation on every two first collected texts after text deduplication by manual or other machine model text comparison methods, and deleting any one of the two first collected texts whose text similarity exceeds a preset similarity threshold;

[0018] A round of traversal is performed on all the remaining first collected texts; and in this round of traversal, the first collected text currently traversed is used as the corresponding current collected text; and whether the current collected text contains extractable relationship triples is identified through the relationship triple recognition method of artificial or other machine models; if there are no extractable relationship triples in the current collected text, the current collected text is deleted; if the current collected text contains extractable relationship triples, the current collected text is used as a corresponding first training text, and each relationship triple in the current collected text is extracted as a corresponding first label relationship triple, and all the extracted first label relationship triples form a corresponding first label relationship triple sequence, and the obtained first training text and the first label relationship triple sequence form a corresponding first data record; and at the end of this round of traversal, all the obtained first data records form the corresponding first data set.

[0019] Preferably, the pre-training of the relationship extraction function of the first model based on the first data set specifically includes:

[0020] Step 401, dividing the first data set into two sub-data sets according to a preset first division ratio and recording them as a corresponding first training set and a first evaluation set;

[0021] Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the first evaluation set satisfies the first segmentation ratio;

[0022] Step 402, taking the first first data record of the first training set as the corresponding current training record;

[0023] Step 403, bringing the first training text of the current training record into a preset first training instruction template to assemble a large model input text to obtain a corresponding first input text; and recording the first label relationship triple sequence of the current training record as the corresponding current label relationship triple sequence;

[0024] The first training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text, and inform the first model through a preset narrative text to perform relation triple recognition on the current text and output all the recognized relation triples as model output text;

[0025] Step 404, inputting the first input text into the first model for processing to obtain a corresponding first output text;

[0026] The first output text includes one or more first relation triples; each of the first relation triples consists of a head entity text, an entity relation text and a tail entity text;

[0027] Step 405, perform a round of traversal on all the first relation triples of the first output text; and in this round of traversal, record the first relation triple currently traversed as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the preset invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding first prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0028] The invalid relationship triple is also composed of a head entity text, an entity relationship text, and a tail entity text, but the head entity text, the entity relationship text, and the tail entity text are all empty texts;

[0029] Step 406: after traversing all the first relation triples of the first output text, identify whether there are any first label relation triples that have not been marked as the used triples in the current label relation triple sequence; if there are any first label relation triples that have not been marked as the used triples, use each of the first label relation triples that have not been marked as the used triples as the corresponding remaining label triples, and use the invalid relation triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding first prediction-label pair by each of the remaining label triples and the corresponding paired prediction triples;

[0030] Step 407, record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset; input all the first prediction-label pairs obtained into a preset first model loss function; and perform a round of parameter optimization on the first function parameter subset in a direction of minimizing the first model loss function based on a preset first model optimizer;

[0031] Among them, the first model loss function includes at least a cross entropy loss function, a logarithmic loss function, a mean square error loss function and a mean absolute error loss function; the first model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer, and an AdamW optimizer;

[0032] Step 408, identifying whether the current training record is the last first data record of the first training set; if so, going to step 409; if not, extracting the next first data record of the first training set as the new current training record and returning to step 403;

[0033] Step 409, taking the first first data record of the first evaluation set as the corresponding current evaluation record;

[0034] Step 410, bringing the first training text of the current evaluation record into the first training instruction template to perform large model input text assembly to obtain a corresponding second input text; and recording the first label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and inputting the second input text into the first model for processing to obtain a corresponding second output text;

[0035] The second output text includes one or more second relation triples; each of the second relation triples consists of a head entity text, an entity relation text and a tail entity text;

[0036] Step 411, perform a round of traversal on all the second relation triples of the second output text; and in this round of traversal, record the second relation triple currently traversed as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding second prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0037] Step 412, after the traversal of all the second relation triples of the second output text is completed, whether there are any first label relation triples that have not been marked as the used triples in the current label relation triple sequence are identified; if there are any first label relation triples that have not been marked as the used triples, then each of the first label relation triples that have not been marked as the used triples is used as the corresponding remaining label triple, and the invalid relation triple is used as the paired prediction triple corresponding to each of the remaining label triples, and each of the remaining label triples and the corresponding paired prediction triple forms a corresponding second prediction-label pair;

[0038] Step 413, identifying whether the current evaluation record is the last first data record of the first evaluation set; if so, going to step 414; if not, extracting the next first data record of the first evaluation set as the new current evaluation record and returning to step 410;

[0039] Step 414, calculating the multiple evaluation parameters of the first model according to all the obtained second prediction-labels to obtain the corresponding first evaluation parameter set; and identifying whether the evaluation parameter values ​​of all the first evaluation parameters of the first evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one of the first evaluation parameters in the first evaluation parameter set does not satisfy the corresponding first parameter value range, returning to step 402 to continue training; if the evaluation parameter values ​​of all the first evaluation parameters satisfy their respective corresponding first parameter value ranges, stopping training and confirming that the relationship extraction function pre-training is completed;

[0040] Among them, the first evaluation parameter set includes multiple first evaluation parameters; the first evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate, F1 score and BLEU score; the evaluation parameter value of each first evaluation parameter has a preset first parameter value range corresponding to it.

[0041] Preferably, the pre-training of the relationship verification function of the first model based on the first data set specifically includes:

[0042] Step 51, dividing the first data set into two sub-data sets according to a preset second division ratio and recording them as corresponding second training sets and second evaluation sets;

[0043] Wherein, both the second training set and the second evaluation set are composed of a plurality of the first data records; the ratio of the total number of records of the second training set to the second evaluation set satisfies the second segmentation ratio;

[0044] Step 52, taking the first of the first data records in the second training set as the corresponding current training record;

[0045] Step 53, bringing the first training text of the current training record and each of the first label relationship triples into a preset second training instruction template to perform large model input text assembly to obtain a corresponding third input text;

[0046] The second training instruction template is a formatted text template, which is used to take the first training text as the corresponding current text, take the first label relationship triplet as the corresponding current relationship triplet, and inform the first model through a preset narrative text whether the association relationship between the head entity and the tail entity reflected by the current relationship triplet in the current text is correctly identified, and output the specific correct or incorrect recognition result as the model output text;

[0047] Step 54, inputting each of the third input texts corresponding to the current training record into the first model for processing to obtain a corresponding third output text;

[0048] Wherein, the third output text includes correct and wrong;

[0049] Step 55, perform a round of traversal on all the third output texts corresponding to the current training record; and in this round of traversal, use the third output text currently traversed as the corresponding current output text; and identify the current output text; if the current output text is correct, set the corresponding first prediction data to 1; if the current output text is wrong, set the corresponding first prediction data to 0; and configure a corresponding first label data for each first prediction data, and set all the first label data to 1; and each first prediction data and the corresponding first label data form a corresponding third prediction-label pair;

[0050] Step 56, record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset, record the model parameter subset related to the relationship verification function as the corresponding second function parameter subset, and record the intersection of the first and second function parameter subsets as the corresponding first parameter intersection, and use the other model parameters in the second function parameter subset except the first parameter intersection as the corresponding second function private parameter subset; input all the third prediction-label pairs corresponding to the current training record into the preset second model loss function; and based on the preset second model optimizer, perform a round of parameter optimization on the second function private parameter subset in the direction of minimizing the second model loss function;

[0051] Wherein, the second model loss function includes at least a mean square error loss function, a mean absolute error loss function and a cross entropy loss function; the second model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer and an AdamW optimizer;

[0052] Step 57, identifying whether the current training record is the last of the first data record in the second training set; if so, going to step 58; if not, extracting the next first data record in the second training set as the new current training record and returning to step 53;

[0053] Step 58, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and bring the first training text and each of the first label relationship triples of the current evaluation record into the second training instruction template for large model input text assembly to obtain a corresponding fourth input text; and input each of the fourth input texts corresponding to the current evaluation record into the first model for processing to obtain a corresponding fourth output text; and identify each of the fourth output texts corresponding to the current evaluation record, and if the current fourth output text is correct, set the corresponding second prediction data to 1, and if the current fourth output text is wrong, set the corresponding second prediction data to 0; and configure a corresponding second label data for each of the second prediction data, and set all the second label data to 1; and each of the second prediction data and the corresponding second label data form a corresponding fourth prediction-label pair;

[0054] Step 59, after the traversal of all the first data records of the second evaluation set is completed, multiple evaluation parameters of the first model are calculated according to all the fourth prediction-labels obtained to obtain a corresponding second evaluation parameter set; and whether the evaluation parameter values ​​of all second evaluation parameters of the second evaluation parameter set satisfy the corresponding second parameter value range is identified; if the evaluation parameter value of at least one second evaluation parameter in the second evaluation parameter set does not satisfy the corresponding second parameter value range, return to step 52 to continue training; if the evaluation parameter values ​​of all the second evaluation parameters satisfy their respective corresponding second parameter value ranges, stop training and confirm that the relationship verification function pre-training is completed;

[0055] Among them, the second evaluation parameter set includes multiple second evaluation parameters; the second evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate and F1 score; the evaluation parameter value of each second evaluation parameter has a preset second parameter value range corresponding to it.

[0056] Preferably, the text generation function based on the first model constructs an enhanced data set, which is recorded as a corresponding second data set, and specifically includes:

[0057] Step 61, initializing the first record counter to 1;

[0058] Step 62, in the preset integer value range [1,n max ] and perform a random rounding to get the corresponding random number N;

[0059] Among them, the maximum value of the range n max , the random number N is a positive integer, 1≤N≤n max ;

[0060] Step 63, bringing the random number N into the preset third training instruction template to assemble the large model input text to obtain a corresponding fifth input text; and inputting the fifth input text into the first model for processing to obtain a corresponding fifth output text;

[0061] The third training instruction template is a formatted text template, which is used to use the random number N as the corresponding current total number of triples x, and inform the first model through a preset narrative text to automatically generate x relationship triples in multiple fields based on all knowledge fields covered by the model itself and output the obtained x relationship triples as model output text; the fifth output text includes N third relationship triples;

[0062] Step 64, bringing the N third relationship triples obtained this time into the preset fourth training instruction template to assemble the large model input text to obtain a corresponding sixth input text; and inputting the sixth input text into the first model for processing to obtain a corresponding sixth output text;

[0063] The fourth training instruction template is a formatted text template, which is used to take the N third relationship triples as the corresponding current triple sequence, and inform the first model through a preset narrative text to automatically generate a creative text based on the current triple sequence as the model output text output, and requires that the creative text content must contain the entity and relationship original text of the current triple sequence;

[0064] Step 65, taking each of the third relation triples obtained this time as a corresponding second label relation triple; and forming a corresponding second label relation triple sequence from the N second label relation triples obtained this time; and taking the sixth output text obtained this time as a corresponding second training text; and forming a corresponding second data record from the second label relation triple sequence obtained this time and the second training text;

[0065] Step 66, adding 1 to the first record counter; and identifying whether the first record counter after adding 1 is greater than the preset maximum total number of records; if not, returning to step 62; if so, all the obtained second data records form the corresponding second data set.

[0066] Preferably, the data cleaning of the second data set based on the relationship verification function of the first model specifically includes:

[0067] A round of traversal is performed on all the second data records of the second data set; and in this round of traversal, the second data record currently traversed is used as the corresponding current data record; and the second training text and each second label relationship triple of the current data record are brought into the second training instruction template for large model input text assembly to obtain a corresponding seventh input text; and each of the seventh input texts corresponding to the current data record is input into the first model for processing to obtain a corresponding seventh output text; and whether all the seventh output texts corresponding to the current data record are correct are identified; if not, the current data record is deleted; wherein, the seventh output text includes correct and incorrect.

[0068] Preferably, the fine-tuning of the relationship extraction function of the first model based on the cleaned second data set specifically includes:

[0069] Step 801, dividing the second data set into two sub-data sets according to a preset third division ratio and recording them as a corresponding third training set and a third evaluation set;

[0070] Wherein, the third training set and the third evaluation set are both composed of a plurality of the second data records; the ratio of the total number of records of the third training set to the third evaluation set satisfies the third segmentation ratio;

[0071] Step 802, taking the first second data record of the third training set as the corresponding current training record;

[0072] Step 803, bringing the second training text of the current training record into the first training instruction template to perform large model input text assembly to obtain the corresponding seventh input text; and recording the second label relationship triple sequence of the current training record as the corresponding current label relationship triple sequence; and inputting the eighth input text into the first model for processing to obtain the corresponding eighth output text;

[0073] The eighth output text includes one or more fourth relation triples; each of the fourth relation triples is composed of a head entity text, an entity relation text and a tail entity text;

[0074] Step 804, perform a round of traversal on all the fourth relationship triples of the eighth output text; and in this round of traversal, record the currently traversed fourth relationship triple as the corresponding current prediction triple; and record the second label relationship triple in the current label relationship triple sequence, whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relationship triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding fifth prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0075] Step 805, after traversing all the fourth relationship triples of the eighth output text, identify whether there are any second label relationship triples that have not been marked as the used triples in the current label relationship triple sequence; if there are any second label relationship triples that have not been marked as the used triples, use each of the second label relationship triples that have not been marked as the used triples as the corresponding remaining label triples, and use the invalid relationship triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding fifth prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples;

[0076] Step 806, input all the fifth prediction-label pairs obtained into the first model loss function; and based on the preset third model optimizer, perform a round of parameter fine-tuning on the first functional parameter subset in the direction of minimizing the first model loss function; and at the end of this round of fine-tuning, identify whether the current training record is the last second data record of the third training set; if so, go to step 807; if not, extract the next second data record of the third training set as the new current training record and return to step 803;

[0077] Wherein, the third model optimizer includes at least ADAM optimizer and AdamW optimizer;

[0078] Step 807, taking the first second data record of the third evaluation set as the corresponding current evaluation record;

[0079] Step 808, bringing the second training text of the current evaluation record into the first training instruction template to assemble the large model input text to obtain the corresponding ninth input text; and recording the second label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and inputting the ninth input text into the first model for processing to obtain the corresponding ninth output text;

[0080] The ninth output text includes one or more fifth relation triples; each of the fifth relation triples is composed of a head entity text, an entity relation text and a tail entity text;

[0081] Step 809, perform a round of traversal on all the fifth relation triples of the ninth output text; and in this round of traversal, record the fifth relation triple currently traversed as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence, whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding sixth prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0082] Step 810, after the traversal of all the fifth relationship triples of the ninth output text is completed, whether there are any second label relationship triples that have not been marked as the used triples in the current label relationship triple sequence are identified; if there are any second label relationship triples that have not been marked as the used triples, then each of the second label relationship triples that have not been marked as the used triples is used as the corresponding remaining label triple, and the invalid relationship triple is used as the paired prediction triple corresponding to each of the remaining label triples, and each of the remaining label triples and the corresponding paired prediction triple forms a corresponding sixth prediction-label pair;

[0083] Step 811, identifying whether the current evaluation record is the last second data record of the third evaluation set; if so, going to step 812; if not, extracting the next second data record of the third evaluation set as the new current evaluation record and returning to step 808;

[0084] Step 812, calculating the multiple evaluation parameters of the first model according to all the obtained sixth prediction-labels to obtain the corresponding third evaluation parameter set; and identifying whether the evaluation parameter values ​​of all the third evaluation parameters of the third evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one of the third evaluation parameters in the third evaluation parameter set does not satisfy the corresponding first parameter value range, returning to step 802 to continue training; if the evaluation parameter values ​​of all the third evaluation parameters satisfy their respective corresponding first parameter value ranges, stopping training and confirming that the fine-tuning of the relationship extraction function is completed;

[0085] The third evaluation parameter set includes a plurality of the third evaluation parameters; the third evaluation parameters include the evaluation parameter type and the evaluation parameter value; and the evaluation parameter value of each of the third evaluation parameters has a corresponding first parameter value range.

[0086] A second aspect of an embodiment of the present invention provides a device for implementing the processing method for performing relationship extraction based on a large language model as described in the first aspect, the device comprising: a model selection module, a model pre-training module, a model fine-tuning module, and a model application module;

[0087] The model selection module is used to select a large language model as the corresponding first model;

[0088] The model pre-training module is used to construct a benchmark label data set as the corresponding first data set through data collection, data cleaning and data annotation by manual or other machine models; and pre-train the relationship extraction function of the first model based on the first data set; and after the relationship extraction function pre-training is completed, the relationship verification function of the first model is pre-trained based on the first data set;

[0089] The model fine-tuning module is used to construct an enhanced data set based on the text generation function of the first model as the corresponding second data set after the pre-training of the relationship verification function is completed; and to clean the second data set based on the relationship verification function of the first model; and to fine-tune the relationship extraction function of the first model based on the cleaned second data set;

[0090] The model application module is used to, after the fine-tuning is completed, bring the first query text, the first restricted relationship sequence and the first output format input by the user into a preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and input the first input text into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user.

[0091] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0092] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0093] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0094] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0095] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for relation extraction based on a large language model. As can be seen from the above content, the embodiment of the present invention selects a class of large language models as the corresponding first model; and constructs a benchmark label data set through data collection, data cleaning and data annotation of artificial or other machine models, which is recorded as the corresponding first data set; and pre-trains the relation extraction function and relation verification function of the first model based on the first data set; and after the pre-training, constructs an enhanced data set based on the text generation function of the first model, which is recorded as the corresponding second data set; and cleans the second data set based on the relation verification function of the first model; and fine-tunes the relation extraction function of the first model based on the cleaned second data set; after the fine-tuning, the first query text, the first limited relation sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model for relation triple recognition and screening to obtain the corresponding first output text to feedback to the user. The embodiment of the present invention utilizes the text generation function of the large language model itself to add a method of constructing an enhanced data set on top of the processing mechanism of a conventional labeled data set, and improves the data accuracy of the enhanced data set by training the relationship verification function of the large language model, and improves the training adequacy and training quality of the model, enhances the generalization of the model, and improves the prediction accuracy of the model through fine-tuning training of the model with the enhanced data set; in addition, the embodiment of the present invention also takes the relationship extraction function of the large language model as the core and provides users with a processing mechanism for identifying and screening relationship triples for specified query texts through a first instruction template. This processing mechanism simplifies the user operation steps and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A schematic diagram of a processing method for performing relationship extraction based on a large language model provided in Embodiment 1 of the present invention;

[0097] Figure 2 A module structure diagram of a processing device for performing relationship extraction based on a large language model provided in Embodiment 2 of the present invention;

[0098] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0100] Embodiment 1 of the present invention provides a processing method for relation extraction based on a large language model, such as Figure 1 As shown in a schematic diagram of a processing method for relation extraction based on a large language model provided in the first embodiment of the present invention, the method mainly includes the following steps:

[0101] Step 1: Select a large language model as the corresponding first model.

[0102] Here, the first model of the embodiment of the present invention includes at least large language models such as the GPT series model, the GPT-RE model, the BERT model, the UIE series model, the QA4RE model, and the Wenxin large model.

[0103] Step 2, constructing a benchmark label data set as the corresponding first data set by data collection, data cleaning and data labeling by manual or other machine models; and pre-training the relationship extraction function of the first model based on the first data set; and after the pre-training of the relationship extraction function is completed, pre-training the relationship verification function of the first model based on the first data set;

[0104] Specifically comprising: step 21, constructing a benchmark label data set as a corresponding first data set through data collection, data cleaning and data labeling by manual or other machine models;

[0105] The first data set includes a plurality of first data records; the first data record includes a first training text and a first label relation triple sequence; the first label relation triple sequence includes one or more first label relation triples (Relational Triple Extraction, RTE); the first label relation triple consists of a head entity (Head Entity / Subject) text, an entity relation (Relation) text, and a tail entity (Tail Entity / Object) text;

[0106] Specifically, it includes: step 211, collecting big data of text information in multiple fields by manual or other machine model data collection methods to obtain corresponding multiple first collected texts;

[0107] Step 212, and perform grammar and content compliance checks on all the first collected texts obtained by manual or other machine model data verification methods, and delete the first collected texts that fail the grammar check or the content compliance check; and perform duplicate text elimination processing on the remaining first collected texts by manual or other machine model text deduplication methods; and perform text similarity calculation on every two first collected texts after text deduplication by manual or other machine model text comparison methods, and delete any one of the two first collected texts whose text similarity exceeds a preset similarity threshold;

[0108] Here, the preset similarity threshold is a preset threshold parameter;

[0109] Step 213, and perform a round of traversal on all the remaining first collected texts; and in this round of traversal, the first collected text currently traversed is used as the corresponding current collected text; and the current collected text is identified by the relationship triple recognition method of artificial or other machine models to determine whether it contains extractable relationship triples; if there are no extractable relationship triples in the current collected text, the current collected text is deleted; if the current collected text contains extractable relationship triples, the current collected text is used as a corresponding first training text, and each relationship triple in the current collected text is extracted as a corresponding first label relationship triple, and all the extracted first label relationship triples form a corresponding first label relationship triple sequence, and the obtained first training text and the first label relationship triple sequence form a corresponding first data record; and at the end of this round of traversal, all the obtained first data records form a corresponding first data set;

[0110] Step 22, pre-training the relationship extraction function of the first model based on the first data set;

[0111] Specifically comprising: step 22-1, dividing the first data set into two sub-data sets according to a preset first division ratio and recording them as a corresponding first training set and a first evaluation set;

[0112] The first segmentation ratio is a preset ratio, such as 8:2; the first training set and the first evaluation set are both composed of a plurality of first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0113] Step 22-2, taking the first first data record of the first training set as the corresponding current training record;

[0114] Step 22-3, bringing the first training text of the current training record into the preset first training instruction template to assemble the large model input text to obtain the corresponding first input text; and recording the first label relationship triple sequence of the current training record as the corresponding current label relationship triple sequence;

[0115] The first training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text, and inform the first model to perform relation triple recognition on the current text through a preset narrative text and output all the recognized relation triples as model output text;

[0116] Step 22-4, inputting the first input text into the first model for processing to obtain a corresponding first output text;

[0117] The first output text includes one or more first relation triples; each first relation triple is composed of a head entity text, an entity relation text and a tail entity text;

[0118] Step 22-5, perform a round of traversal on all first relation triples of the first output text; and in this round of traversal, record the currently traversed first relation triple as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the preset invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding first prediction-label pair by the current prediction triple and the corresponding pairing label triple;

[0119] Among them, the invalid relation triple is also composed of the head entity text, entity relation text and tail entity text, but its head entity text, entity relation text and tail entity text are all empty text;

[0120] Here, the current step is actually to configure a paired label triple for each predicted first relation triple. The configuration method of this label triple is: query whether there is a matching first label relation triple (with the head entity as the alignment matching relationship) in the corresponding first label relation triple sequence. If there is, the matching first label relation triple is used as the label triple of the current first relation triple. If not, the label triple of the current first relation triple is set to a completely empty invalid relation triple.

[0121] Step 22-6, after the traversal of all first relation triples of the first output text is completed, whether there are any first label relation triples that have not been marked as used triples in the current label relation triple sequence are identified; if there are any first label relation triples that have not been marked as used triples, then each first label relation triple that has not been marked as used triple is used as the corresponding remaining label triple, and the invalid relation triple is used as the paired prediction triple corresponding to each remaining label triple, and each remaining label triple and the corresponding paired prediction triple form a corresponding first prediction-label pair;

[0122] Here, the current step is actually to configure a completely empty invalid relation triple as a paired predicted triple for the unpredicted relation triple, i.e., the first label relation triple in the first label relation triple sequence that is not marked as a used triple;

[0123] Step 22-7, recording the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset; inputting all the obtained first prediction-label pairs into the preset first model loss function; and performing a round of parameter optimization on the first function parameter subset in the direction of minimizing the first model loss function based on the preset first model optimizer;

[0124] Among them, the first model loss function at least includes a cross entropy loss function, a logarithmic loss function, a mean square error loss function and a mean absolute error loss function; the first model optimizer at least includes an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer, and an AdamW optimizer;

[0125] Step 22-8, identifying whether the current training record is the last first data record of the first training set; if so, go to step 22-9; if not, extract the next first data record of the first training set as the new current training record and return to step 22-3;

[0126] Step 22-9, taking the first first data record of the first evaluation set as the corresponding current evaluation record;

[0127] Step 22-10, bringing the first training text of the current evaluation record into the first training instruction template to assemble the large model input text to obtain the corresponding second input text; and recording the first label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and inputting the second input text into the first model for processing to obtain the corresponding second output text;

[0128] The second output text includes one or more second relation triples; each second relation triple is composed of a head entity text, an entity relation text and a tail entity text;

[0129] Step 22-11, perform a round of traversal on all second relation triples of the second output text; and in this round of traversal, record the currently traversed second relation triple as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence, whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding second prediction-label pair by the current prediction triple and the corresponding pairing label triple;

[0130] Step 22-12, after the traversal of all second relation triples of the second output text is completed, whether there are any first label relation triples that have not been marked as used triples in the current label relation triple sequence are identified; if there are any first label relation triples that have not been marked as used triples, then each first label relation triple that has not been marked as used triple is used as the corresponding remaining label triple, and the invalid relation triple is used as the paired prediction triple corresponding to each remaining label triple, and each remaining label triple and the corresponding paired prediction triple form a corresponding second prediction-label pair;

[0131] Step 22-13, identifying whether the current evaluation record is the last first data record of the first evaluation set; if so, go to step 22-14; if not, extract the next first data record of the first evaluation set as the new current evaluation record and return to step 22-10;

[0132] Step 22-14, calculating the multiple evaluation parameters of the first model according to all the obtained second prediction-labels to obtain the corresponding first evaluation parameter set; and identifying whether the evaluation parameter values ​​of all the first evaluation parameters in the first evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one first evaluation parameter in the first evaluation parameter set does not satisfy the corresponding first parameter value range, returning to step 22-2 to continue training; if the evaluation parameter values ​​of all the first evaluation parameters satisfy their respective corresponding first parameter value ranges, stopping the training and confirming that the relationship extraction function pre-training is completed;

[0133] Wherein, the first evaluation parameter set includes multiple first evaluation parameters; the first evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall, F1 score and BLEU score; the evaluation parameter value of each first evaluation parameter has a preset first parameter value range corresponding thereto;

[0134] Step 23, after the pre-training of the relationship extraction function is completed, the relationship verification function of the first model is pre-trained based on the first data set;

[0135] Specifically, it includes: step 231, dividing the first data set into two sub-data sets according to a preset second division ratio and recording them as corresponding second training sets and second evaluation sets;

[0136] The second segmentation ratio is a preset ratio, such as 8:2; the second training set and the second evaluation set are both composed of a plurality of first data records; the ratio of the total number of records in the second training set to the second evaluation set satisfies the second segmentation ratio;

[0137] Step 232, taking the first first data record of the second training set as the corresponding current training record;

[0138] Step 233, bringing the first training text of the current training record and each first label relationship triple into the preset second training instruction template to assemble the large model input text to obtain a corresponding third input text;

[0139] The second training instruction template is a formatted text template, which is used to take the first training text as the corresponding current text, take the first label relationship triplet as the corresponding current relationship triplet, and inform the first model through a preset narrative text whether the association relationship between the head entity and the tail entity reflected by the current relationship triplet in the current text is correctly identified and the specific correct or incorrect recognition result is output as the model output text;

[0140] Step 234, input each third input text corresponding to the current training record into the first model for processing to obtain a corresponding third output text;

[0141] Wherein, the third output text includes correct and wrong;

[0142] Step 235, perform a round of traversal on all third output texts corresponding to the current training record; and in this round of traversal, use the third output text currently traversed as the corresponding current output text; and recognize the current output text; if the current output text is correct, set the corresponding first prediction data to 1; if the current output text is wrong, set the corresponding first prediction data to 0; and configure a corresponding first label data for each first prediction data, and set all first label data to 1; and form a corresponding third prediction-label pair by each first prediction data and the corresponding first label data;

[0143] Step 236, record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset, record the model parameter subset related to the relationship verification function as the corresponding second function parameter subset, record the intersection of the first and second function parameter subsets as the corresponding first parameter intersection, and use the other model parameters in the second function parameter subset except the first parameter intersection as the corresponding second function private parameter subset; input all third prediction-label pairs corresponding to the current training record into the preset second model loss function; and perform a round of parameter optimization on the second function private parameter subset in the direction of minimizing the second model loss function based on the preset second model optimizer;

[0144] Among them, the second model loss function at least includes a mean square error loss function, a mean absolute error loss function and a cross entropy loss function; the second model optimizer at least includes an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer, and an AdamW optimizer;

[0145] Step 237, identifying whether the current training record is the last first data record of the second training set; if so, go to step 238; if not, extract the next first data record of the second training set as the new current training record and return to step 233;

[0146] Step 238, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the currently traversed first data record as the corresponding current evaluation record; and bring the first training text and each first label relationship triple of the current evaluation record into the second training instruction template for large model input text assembly to obtain a corresponding fourth input text; and input each fourth input text corresponding to the current evaluation record into the first model for processing to obtain a corresponding fourth output text; and identify each fourth output text corresponding to the current evaluation record, if the current fourth output text is correct, set the corresponding second prediction data to 1, if the current fourth output text is wrong, set the corresponding second prediction data to 0; and configure a corresponding second label data for each second prediction data, and set all second label data to 1; and each second prediction data and the corresponding second label data form a corresponding fourth prediction-label pair;

[0147] Step 239, after the traversal of all the first data records of the second evaluation set is completed, the multiple evaluation parameters of the first model are calculated according to all the fourth prediction-labels obtained to obtain the corresponding second evaluation parameter set; and whether the evaluation parameter values ​​of all the second evaluation parameters of the second evaluation parameter set satisfy the corresponding second parameter value range is identified; if the evaluation parameter value of at least one second evaluation parameter in the second evaluation parameter set does not satisfy the corresponding second parameter value range, return to step 232 to continue training; if the evaluation parameter values ​​of all the second evaluation parameters satisfy their respective corresponding second parameter value ranges, stop training and confirm that the relationship verification function pre-training is completed;

[0148] Among them, the second evaluation parameter set includes multiple second evaluation parameters; the second evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate and F1 score; the evaluation parameter value of each second evaluation parameter has a preset second parameter value range corresponding to it.

[0149] Step 3: After the pre-training of the relationship verification function is completed, an enhanced data set is constructed based on the text generation function of the first model and recorded as the corresponding second data set; and the second data set is cleaned based on the relationship verification function of the first model; and the relationship extraction function of the first model is fine-tuned based on the cleaned second data set;

[0150] Specifically, it includes: step 31, after the pre-training of the relationship verification function is completed, an enhanced data set is constructed based on the text generation function of the first model and recorded as the corresponding second data set;

[0151] The second data set includes a plurality of second data records; the second data record includes a second label relationship triple sequence and a second training text; the second label relationship triple sequence includes one or more second label relationship triples; the second label relationship triple consists of a head entity text, an entity relationship text and a tail entity text;

[0152] Specifically comprising: step 311, initializing a first record counter to 1;

[0153] Step 312: In the preset integer range [1,n max ] and perform a random rounding to get the corresponding random number N;

[0154] Among them, the maximum value of the range n max , the random number N is a positive integer, 1≤N≤n max ;

[0155] Step 313, bringing the random number N into the preset third training instruction template to assemble the large model input text to obtain a corresponding fifth input text; and inputting the fifth input text into the first model for processing to obtain a corresponding fifth output text;

[0156] The third training instruction template is a formatted text template, which is used to take the random number N as the corresponding total number of current triples x, and inform the first model through a preset narrative text to automatically generate x relationship triples in multiple fields based on all knowledge fields covered by the model itself and output the obtained x relationship triples as model output text; the fifth output text includes N third relationship triples;

[0157] Here, the third training instruction template of the embodiment of the present invention utilizes the knowledge domain scope of the first model itself to improve the domain width of the data;

[0158] Step 314, bringing the N third relationship triples obtained this time into the preset fourth training instruction template to assemble the large model input text to obtain a corresponding sixth input text; and inputting the sixth input text into the first model for processing to obtain a corresponding sixth output text;

[0159] The fourth training instruction template is a formatted text template, which is used to take N third-relation triples as the corresponding current triple sequence, and inform the first model through a preset narrative text to automatically generate a creative text based on the current triple sequence as the model output text output, and requires that the creative text content must contain the entity and relationship original text of the current triple sequence;

[0160] Here, the fourth training instruction template of the embodiment of the present invention utilizes the text generation capability of the first model itself to achieve the purpose of automatically generating enhanced data;

[0161] Step 315, each third relation triple obtained this time is used as a corresponding second label relation triple; and the N second label relation triples obtained this time form a corresponding second label relation triple sequence; and the sixth output text obtained this time is used as a corresponding second training text; and the second label relation triple sequence obtained this time and the second training text form a corresponding second data record;

[0162] Step 316, adding 1 to the first record counter; and identifying whether the first record counter after adding 1 is greater than the preset maximum total number of records; if not, returning to step 312; if so, forming a corresponding second data set from all the obtained second data records;

[0163] Here, the maximum number of records is a preset large integer, which can be set based on the training sufficiency requirement; considering the maximum number of records as a large integer, a large number of enhanced data sets with higher data quality and spanning all model knowledge domains, i.e., the second data set, can be obtained;

[0164] Step 32, performing data cleaning on the second data set based on the relationship verification function of the first model;

[0165] Specifically, the method includes: performing a round of traversal on all second data records of the second data set; and in the process of this round of traversal, taking the second data record currently traversed as the corresponding current data record; and bringing the second training text and each second label relationship triple of the current data record into the second training instruction template to perform large model input text assembly to obtain a corresponding seventh input text; and inputting each seventh input text corresponding to the current data record into the first model for processing to obtain a corresponding seventh output text; and identifying whether all the seventh output texts corresponding to the current data record are correct; if not, deleting the current data record;

[0166] Wherein, the seventh output text includes correct and wrong;

[0167] Here, the embodiment of the present invention further improves the data accuracy of the second data set by using the relationship verification function of the first model;

[0168] Step 33, fine-tuning the relationship extraction function of the first model based on the cleaned second data set;

[0169] Specifically comprising: step 33-1, dividing the second data set into two sub-data sets according to a preset third division ratio and recording them as a corresponding third training set and a third evaluation set;

[0170] The third segmentation ratio is a preset ratio, such as 8:2; the third training set and the third evaluation set are both composed of a plurality of second data records; the ratio of the total number of records of the third training set to the third evaluation set satisfies the third segmentation ratio;

[0171] Step 33-2, taking the first second data record of the third training set as the corresponding current training record;

[0172] Step 33-3, bringing the second training text of the current training record into the first training instruction template to assemble the large model input text to obtain the corresponding seventh input text; and recording the second label relationship triple sequence of the current training record as the corresponding current label relationship triple sequence; and inputting the eighth input text into the first model for processing to obtain the corresponding eighth output text;

[0173] The eighth output text includes one or more fourth relation triples; each fourth relation triple is composed of a head entity text, an entity relation text, and a tail entity text;

[0174] Step 33-4, perform a round of traversal on all fourth relation triples of the eighth output text; and in this round of traversal, record the currently traversed fourth relation triple as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence, whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding fifth prediction-label pair by the current prediction triple and the corresponding pairing label triple;

[0175] Step 33-5, after the traversal of all fourth relationship triples of the eighth output text is completed, whether there are any second label relationship triples that have not been marked as used triples in the current label relationship triple sequence are identified; if there are any second label relationship triples that have not been marked as used triples, then each second label relationship triple that has not been marked as a used triple is used as the corresponding remaining label triple, and the invalid relationship triple is used as the paired prediction triple corresponding to each remaining label triple, and each remaining label triple and the corresponding paired prediction triple form a corresponding fifth prediction-label pair;

[0176] Step 33-6, input all the fifth prediction-label pairs obtained into the first model loss function; and based on the preset third model optimizer, perform a round of parameter fine-tuning on the first functional parameter subset in the direction of minimizing the first model loss function; and at the end of this round of fine-tuning, identify whether the current training record is the last second data record of the third training set; if so, go to step 33-7; if not, extract the next second data record of the third training set as the new current training record and return to step 33-3;

[0177] Wherein, the third model optimizer includes at least an ADAM optimizer and an AdamW optimizer;

[0178] Step 33-7, taking the first second data record of the third evaluation set as the corresponding current evaluation record;

[0179] Step 33-8, bringing the second training text of the current evaluation record into the first training instruction template to assemble the large model input text to obtain the corresponding ninth input text; and recording the second label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and inputting the ninth input text into the first model for processing to obtain the corresponding ninth output text;

[0180] The ninth output text includes one or more fifth relation triples; each fifth relation triple is composed of a head entity text, an entity relation text, and a tail entity text;

[0181] Step 33-9, perform a round of traversal on all fifth relation triples of the ninth output text; and in this round of traversal, record the fifth relation triple currently traversed as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence, whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding sixth prediction-label pair by the current prediction triple and the corresponding pairing label triple;

[0182] Step 33-10, after the traversal of all fifth relation triples of the ninth output text is completed, whether there are any second label relation triples that have not been marked as used triples in the current label relation triple sequence are identified; if there are any second label relation triples that have not been marked as used triples, then each second label relation triple that has not been marked as used triple is used as the corresponding remaining label triple, and the invalid relation triple is used as the paired prediction triple corresponding to each remaining label triple, and each remaining label triple and the corresponding paired prediction triple form a corresponding sixth prediction-label pair;

[0183] Step 33-11, identifying whether the current evaluation record is the last second data record of the third evaluation set; if so, go to step 33-12; if not, extract the next second data record of the third evaluation set as the new current evaluation record and return to step 33-8;

[0184] Step 33-12, calculating the multiple evaluation parameters of the first model according to all the obtained sixth prediction-labels to obtain the corresponding third evaluation parameter set; and identifying whether the evaluation parameter values ​​of all the third evaluation parameters of the third evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one third evaluation parameter in the third evaluation parameter set does not satisfy the corresponding first parameter value range, returning to step 33-2 to continue training; if the evaluation parameter values ​​of all the third evaluation parameters satisfy their respective corresponding first parameter value ranges, stopping training and confirming that the fine-tuning of the relationship extraction function is completed;

[0185] Among them, the third evaluation parameter set includes multiple third evaluation parameters; the third evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate, F1 score and BLEU score; the evaluation parameter value of each third evaluation parameter has a corresponding first parameter value range.

[0186] Step 4: After fine-tuning is completed, the first query text, the first restricted relation sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user;

[0187] Specifically, it includes: step 41, after the fine-tuning is completed, the first query text, the first limiting relationship sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text;

[0188] Wherein, the first restricted relationship sequence is composed of one or more first entity relationship texts;

[0189] The first output format includes at least an array sequence format with a head entity-entity relationship-tail entity field, a list format with a head entity-entity relationship-tail entity field, and a specified script format with a head entity-entity relationship-tail entity field; the specified script format includes at least a BAT script, an HTM script, a JSON script, and a Python script;

[0190] The first instruction template is a formatted text template, which is used to use the first query text, the first restricted relationship sequence and the first output format as the corresponding current query text, the current restricted relationship sequence and the current output format, and inform the large language model through a preset narrative text to extract relation triples from the current query text, and filter all extracted relation triples based on the current restricted relationship sequence, and assemble and output the triple relations that meet the current restricted relationship sequence according to the current output format;

[0191] Step 42, the first input text is input into the first model for relation triple recognition and screening to obtain a corresponding first output text to be fed back to the user.

[0192] For example, it is known that the first model is the Wenxinda model;

[0193] The text format of the first instruction template is:

[0194] "Current query text = <>,

[0195] Current limiting relation sequence = <>,

[0196] Current output format = <>,

[0197] Please extract relation triples from the current query text, filter all extracted relation triples based on the current restricted relation sequence, and assemble and output the triple relations that meet the current restricted relation sequence in the current output format.

[0198] The first query text "Narrative prose refers to prose that is mainly about recording people and narrating stories, and is filled with a strong lyrical atmosphere; it features specific descriptions and descriptions of people and things, and at the same time expresses the author's understanding and feelings", the first limiting relationship sequence "features" and the first output format "JSON script format with head entity-entity relationship-tail entity field" are brought into the first instruction template to obtain the first input text:

[0199] "Current query text = <Narrative prose refers to prose that is mainly about recording people and narrating stories, and is filled with a strong lyrical atmosphere; it is characterized by the specific description and description of people and things, and at the same time expresses the author's understanding and feelings>,

[0200] Current qualified relation sequence = <feature>,

[0201] Current output format = <JSON script format with header entity-entity relationship-footer entity fields>,

[0202] Please extract relation triples from the current query text, filter all extracted relation triples based on the current restricted relation sequence, and assemble and output the triple relations that meet the current restricted relation sequence in the current output format.

[0203] The JSON script obtained by inputting the first input text into the first model is:

[0204]

[0205] Figure 2 The module structure diagram of a processing device for relation extraction based on a large language model provided in the second embodiment of the present invention, the device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment, for example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a model selection module 201, a model pre-training module 202, a model fine-tuning module 203 and a model application module 204.

[0206] The model selection module 201 is used to select a large language model as the corresponding first model.

[0207] The model pre-training module 202 is used to construct a benchmark label data set as the corresponding first data set through data collection, data cleaning and data annotation of manual or other machine models; and pre-train the relationship extraction function of the first model based on the first data set; and after the pre-training of the relationship extraction function is completed, pre-train the relationship verification function of the first model based on the first data set.

[0208] The model fine-tuning module 203 is used to construct an enhanced data set based on the text generation function of the first model as the corresponding second data set after the pre-training of the relationship verification function is completed; and to clean the second data set based on the relationship verification function of the first model; and to fine-tune the relationship extraction function of the first model based on the cleaned second data set.

[0209] The model application module 204 is used to, after the fine-tuning is completed, bring the first query text, the first restricted relationship sequence and the first output format input by the user into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and input the first input text into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text to feedback to the user.

[0210] An embodiment of the present invention provides a processing device for relationship extraction based on a large language model, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0211] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the model selection module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0212] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0213] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0214] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0215] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0216] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0217] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0218] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for relation extraction based on a large language model. As can be seen from the above content, the embodiment of the present invention selects a class of large language models as the corresponding first model; and constructs a benchmark label data set through data collection, data cleaning and data annotation of artificial or other machine models, which is recorded as the corresponding first data set; and pre-trains the relation extraction function and relation verification function of the first model based on the first data set; and after the pre-training, constructs an enhanced data set based on the text generation function of the first model, which is recorded as the corresponding second data set; and cleans the second data set based on the relation verification function of the first model; and fine-tunes the relation extraction function of the first model based on the cleaned second data set; after the fine-tuning, the first query text, the first limited relation sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model for relation triple recognition and screening to obtain the corresponding first output text to feedback to the user. The embodiment of the present invention utilizes the text generation function of the large language model itself to add a method of constructing an enhanced data set on top of the processing mechanism of a conventional labeled data set, and improves the data accuracy of the enhanced data set by training the relationship verification function of the large language model, and improves the training adequacy and training quality of the model, enhances the generalization of the model, and improves the prediction accuracy of the model through fine-tuning training of the model with the enhanced data set; in addition, the embodiment of the present invention also takes the relationship extraction function of the large language model as the core and provides users with a processing mechanism for identifying and screening relationship triples for specified query texts through a first instruction template. This processing mechanism simplifies the user operation steps and improves the user experience.

[0219] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0220] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for relation extraction based on a large language model, characterized in that: The method comprises: Select a class of large language models as the corresponding first model; A benchmark label data set is constructed by data collection, data cleaning and data labeling by manual or other machine models, and recorded as the corresponding first data set; and a relationship extraction function of the first model is pre-trained based on the first data set; and after the relationship extraction function pre-training is completed, a relationship verification function of the first model is pre-trained based on the first data set; After the pre-training of the relationship verification function is completed, an enhanced data set is constructed based on the text generation function of the first model and recorded as the corresponding second data set; and the second data set is cleaned based on the relationship verification function of the first model; and the relationship extraction function of the first model is fine-tuned based on the cleaned second data set; After fine-tuning is completed, the first query text, the first restricted relationship sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text to be fed back to the user.

2. The method for relation extraction based on a large language model according to claim 1, characterized in that: The first model includes at least a GPT series model, a GPT-RE model, a BERT model, a UIE series model, a QA4RE model, and a Wenxin model; The first data set includes a plurality of first data records; the first data record includes a first training text and a first label relationship triple sequence; the first label relationship triple sequence includes one or more first label relationship triples; The first label relationship triplet consists of a head entity text, an entity relationship text and a tail entity text; The second data set includes a plurality of second data records; the second data record includes a second label relation triple sequence and a second training text; The second label relation triple sequence includes one or more second label relation triples; The second label relationship triplet consists of a head entity text, an entity relationship text and a tail entity text; The first restricted relationship sequence consists of one or more first entity relationship texts; The first output format at least includes an array sequence format with a head entity-entity relationship-tail entity field, a list format with a head entity-entity relationship-tail entity field, and a specified script format with a head entity-entity relationship-tail entity field; the specified script format at least includes a BAT script, an HTM script, a JSON script, and a Python script; The first instruction template is a formatted text template, which is used to use the first query text, the first restricted relationship sequence and the first output format as the corresponding current query text, the current restricted relationship sequence and the current output format, and to inform the large language model through a preset narrative text to extract relation triples from the current query text, and to filter all extracted relation triples based on the current restricted relationship sequence, and to assemble and output the output text according to the current output format for the triple relationships that satisfy the current restricted relationship sequence.

3. The method for relation extraction based on a large language model according to claim 2, characterized in that: The benchmark label data set constructed by artificial or other machine model data collection, data cleaning and data labeling is recorded as the corresponding first data set, specifically including: Performing big data collection of text information in multiple fields by manual or other machine model data collection methods to obtain corresponding multiple first collected texts; and performing grammar and content compliance checks on all the first collected texts obtained by manual or other machine model data verification methods, and deleting the first collected texts that fail the grammar check or the content compliance check; and performing duplicate text elimination processing on the remaining first collected texts by manual or other machine model text deduplication methods; and performing text similarity calculation on every two first collected texts after text deduplication by manual or other machine model text comparison methods, and deleting any one of the two first collected texts whose text similarity exceeds a preset similarity threshold; A round of traversal is performed on all the remaining first collected texts; and in this round of traversal, the first collected text currently traversed is used as the corresponding current collected text; and whether the current collected text contains extractable relationship triples is identified through the relationship triple recognition method of artificial or other machine models; if there are no extractable relationship triples in the current collected text, the current collected text is deleted; if the current collected text contains extractable relationship triples, the current collected text is used as a corresponding first training text, and each relationship triple in the current collected text is extracted as a corresponding first label relationship triple, and all the extracted first label relationship triples form a corresponding first label relationship triple sequence, and the obtained first training text and the first label relationship triple sequence form a corresponding first data record; and at the end of this round of traversal, all the obtained first data records form the corresponding first data set.

4. The method for relation extraction based on a large language model according to claim 2, characterized in that: The pre-training of the relationship extraction function of the first model based on the first data set specifically includes: Step 401, dividing the first data set into two sub-data sets according to a preset first division ratio and recording them as a corresponding first training set and a first evaluation set; Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the first evaluation set satisfies the first segmentation ratio; Step 402, taking the first first data record of the first training set as the corresponding current training record; Step 403, bringing the first training text of the current training record into a preset first training instruction template to assemble a large model input text to obtain a corresponding first input text; and recording the first label relationship triple sequence of the current training record as the corresponding current label relationship triple sequence; The first training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text, and inform the first model through a preset narrative text to perform relation triple recognition on the current text and output all the recognized relation triples as model output text; Step 404, inputting the first input text into the first model for processing to obtain a corresponding first output text; The first output text includes one or more first relation triples; each of the first relation triples consists of a head entity text, an entity relation text and a tail entity text; Step 405, perform a round of traversal on all the first relation triples of the first output text; and in this round of traversal, record the first relation triple currently traversed as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the preset invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding first prediction-label pair by the current prediction triple and the corresponding paired label triple; The invalid relationship triple is also composed of a head entity text, an entity relationship text, and a tail entity text, but the head entity text, the entity relationship text, and the tail entity text are all empty texts; Step 406: after traversing all the first relation triples of the first output text, identify whether there are any first label relation triples that have not been marked as the used triples in the current label relation triple sequence; if there are any first label relation triples that have not been marked as the used triples, use each of the first label relation triples that have not been marked as the used triples as the corresponding remaining label triples, and use the invalid relation triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding first prediction-label pair by each of the remaining label triples and the corresponding paired prediction triples; Step 407, record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset; input all the first prediction-label pairs obtained into a preset first model loss function; and perform a round of parameter optimization on the first function parameter subset in a direction of minimizing the first model loss function based on a preset first model optimizer; Among them, the first model loss function includes at least a cross entropy loss function, a logarithmic loss function, a mean square error loss function and a mean absolute error loss function; the first model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer, and an AdamW optimizer; Step 408, identifying whether the current training record is the last first data record of the first training set; if so, going to step 409; if not, extracting the next first data record of the first training set as the new current training record and returning to step 403; Step 409, taking the first first data record of the first evaluation set as the corresponding current evaluation record; Step 410, bringing the first training text of the current evaluation record into the first training instruction template to perform large model input text assembly to obtain a corresponding second input text; and recording the first label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and inputting the second input text into the first model for processing to obtain a corresponding second output text; The second output text includes one or more second relation triples; each of the second relation triples consists of a head entity text, an entity relation text and a tail entity text; Step 411, perform a round of traversal on all the second relation triples of the second output text; and in this round of traversal, record the second relation triple currently traversed as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding second prediction-label pair by the current prediction triple and the corresponding paired label triple; Step 412, after the traversal of all the second relation triples of the second output text is completed, whether there are any first label relation triples that have not been marked as the used triples in the current label relation triple sequence are identified; if there are any first label relation triples that have not been marked as the used triples, then each of the first label relation triples that have not been marked as the used triples is used as the corresponding remaining label triple, and the invalid relation triple is used as the paired prediction triple corresponding to each of the remaining label triples, and each of the remaining label triples and the corresponding paired prediction triple forms a corresponding second prediction-label pair; Step 413, identifying whether the current evaluation record is the last first data record of the first evaluation set; if so, going to step 414; if not, extracting the next first data record of the first evaluation set as the new current evaluation record and returning to step 410; Step 414, calculating the multiple evaluation parameters of the first model according to all the obtained second prediction-labels to obtain the corresponding first evaluation parameter set; and identifying whether the evaluation parameter values ​​of all the first evaluation parameters of the first evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one of the first evaluation parameters in the first evaluation parameter set does not satisfy the corresponding first parameter value range, returning to step 402 to continue training; if the evaluation parameter values ​​of all the first evaluation parameters satisfy their respective corresponding first parameter value ranges, stopping training and confirming that the relationship extraction function pre-training is completed; Among them, the first evaluation parameter set includes multiple first evaluation parameters; the first evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate, F1 score and BLEU score; the evaluation parameter value of each first evaluation parameter has a preset first parameter value range corresponding to it.

5. The method for relation extraction based on a large language model according to claim 2, characterized in that: The pre-training of the relationship verification function of the first model based on the first data set specifically includes: Step 51, dividing the first data set into two sub-data sets according to a preset second division ratio and recording them as corresponding second training sets and second evaluation sets; Wherein, both the second training set and the second evaluation set are composed of a plurality of the first data records; the ratio of the total number of records of the second training set to the second evaluation set satisfies the second segmentation ratio; Step 52, taking the first of the first data records in the second training set as the corresponding current training record; Step 53, bringing the first training text of the current training record and each of the first label relationship triples into a preset second training instruction template to perform large model input text assembly to obtain a corresponding third input text; The second training instruction template is a formatted text template, which is used to take the first training text as the corresponding current text, take the first label relationship triplet as the corresponding current relationship triplet, and inform the first model through a preset narrative text whether the association relationship between the head entity and the tail entity reflected by the current relationship triplet in the current text is correctly identified, and output the specific correct or incorrect recognition result as the model output text; Step 54, inputting each of the third input texts corresponding to the current training record into the first model for processing to obtain a corresponding third output text; Wherein, the third output text includes correct and wrong; Step 55, perform a round of traversal on all the third output texts corresponding to the current training record; and in this round of traversal, use the third output text currently traversed as the corresponding current output text; and identify the current output text; if the current output text is correct, set the corresponding first prediction data to 1; if the current output text is wrong, set the corresponding first prediction data to 0; and configure a corresponding first label data for each first prediction data, and set all the first label data to 1; and each first prediction data and the corresponding first label data form a corresponding third prediction-label pair; Step 56, record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset, record the model parameter subset related to the relationship verification function as the corresponding second function parameter subset, and record the intersection of the first and second function parameter subsets as the corresponding first parameter intersection, and use the other model parameters in the second function parameter subset except the first parameter intersection as the corresponding second function private parameter subset; input all the third prediction-label pairs corresponding to the current training record into the preset second model loss function; and based on the preset second model optimizer, perform a round of parameter optimization on the second function private parameter subset in the direction of minimizing the second model loss function; Wherein, the second model loss function includes at least a mean square error loss function, a mean absolute error loss function and a cross entropy loss function; the second model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer and an AdamW optimizer; Step 57, identifying whether the current training record is the last of the first data record in the second training set; if so, going to step 58; if not, extracting the next first data record in the second training set as the new current training record and returning to step 53; Step 58, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the first data record currently traversed as the corresponding current evaluation record; and bring the first training text and each of the first label relationship triples of the current evaluation record into the second training instruction template for large model input text assembly to obtain a corresponding fourth input text; and input each of the fourth input texts corresponding to the current evaluation record into the first model for processing to obtain a corresponding fourth output text; and identify each of the fourth output texts corresponding to the current evaluation record, and if the current fourth output text is correct, set the corresponding second prediction data to 1, and if the current fourth output text is wrong, set the corresponding second prediction data to 0; and configure a corresponding second label data for each of the second prediction data, and set all the second label data to 1; and each of the second prediction data and the corresponding second label data form a corresponding fourth prediction-label pair; Step 59, after the traversal of all the first data records of the second evaluation set is completed, multiple evaluation parameters of the first model are calculated according to all the fourth prediction-labels obtained to obtain a corresponding second evaluation parameter set; and whether the evaluation parameter values ​​of all second evaluation parameters of the second evaluation parameter set satisfy the corresponding second parameter value range is identified; if the evaluation parameter value of at least one second evaluation parameter in the second evaluation parameter set does not satisfy the corresponding second parameter value range, return to step 52 to continue training; if the evaluation parameter values ​​of all the second evaluation parameters satisfy their respective corresponding second parameter value ranges, stop training and confirm that the relationship verification function pre-training is completed; Among them, the second evaluation parameter set includes multiple second evaluation parameters; the second evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate and F1 score; the evaluation parameter value of each second evaluation parameter has a preset second parameter value range corresponding to it.

6. The method for relation extraction based on a large language model according to claim 2, characterized in that: The enhanced data set constructed based on the text generation function of the first model is recorded as the corresponding second data set, which specifically includes: Step 61, initializing the first record counter to 1; Step 62, in the preset integer value range [1,n max ] and perform a random rounding to get the corresponding random number N; Among them, the maximum value of the range n max , the random number N is a positive integer, 1≤N≤n max ; Step 63, bringing the random number N into the preset third training instruction template to assemble the large model input text to obtain a corresponding fifth input text; and inputting the fifth input text into the first model for processing to obtain a corresponding fifth output text; The third training instruction template is a formatted text template, which is used to use the random number N as the corresponding current total number of triples x, and inform the first model through a preset narrative text to automatically generate x relationship triples in multiple fields based on all knowledge fields covered by the model itself and output the obtained x relationship triples as model output text; the fifth output text includes N third relationship triples; Step 64, bringing the N third relationship triples obtained this time into the preset fourth training instruction template to assemble the large model input text to obtain a corresponding sixth input text; and inputting the sixth input text into the first model for processing to obtain a corresponding sixth output text; The fourth training instruction template is a formatted text template, which is used to take the N third relationship triples as the corresponding current triple sequence, and inform the first model through a preset narrative text to automatically generate a creative text based on the current triple sequence as the model output text output, and requires that the creative text content must contain the entity and relationship original text of the current triple sequence; Step 65, taking each of the third relation triples obtained this time as a corresponding second label relation triple; and forming a corresponding second label relation triple sequence from the N second label relation triples obtained this time; and taking the sixth output text obtained this time as a corresponding second training text; and forming a corresponding second data record from the second label relation triple sequence obtained this time and the second training text; Step 66, adding 1 to the first record counter; and identifying whether the first record counter after adding 1 is greater than the preset maximum total number of records; if not, returning to step 62; if so, all the obtained second data records form the corresponding second data set.

7. The method for relation extraction based on a large language model according to claim 5, characterized in that: The performing data cleaning on the second data set based on the relationship verification function of the first model specifically includes: A round of traversal is performed on all the second data records of the second data set; and in this round of traversal, the second data record currently traversed is used as the corresponding current data record; and the second training text and each second label relationship triple of the current data record are brought into the second training instruction template for large model input text assembly to obtain a corresponding seventh input text; and each of the seventh input texts corresponding to the current data record is input into the first model for processing to obtain a corresponding seventh output text; and whether all the seventh output texts corresponding to the current data record are correct are identified; if not, the current data record is deleted; wherein, the seventh output text includes correct and incorrect.

8. The method for relation extraction based on a large language model according to claim 4, characterized in that: The fine-tuning of the relationship extraction function of the first model based on the cleaned second data set specifically includes: Step 801, dividing the second data set into two sub-data sets according to a preset third division ratio and recording them as a corresponding third training set and a third evaluation set; Wherein, the third training set and the third evaluation set are both composed of a plurality of the second data records; the ratio of the total number of records of the third training set to the third evaluation set satisfies the third segmentation ratio; Step 802, taking the first second data record of the third training set as the corresponding current training record; Step 803, bringing the second training text of the current training record into the first training instruction template to perform large model input text assembly to obtain the corresponding seventh input text; and recording the second label relationship triple sequence of the current training record as the corresponding current label relationship triple sequence; and inputting the eighth input text into the first model for processing to obtain the corresponding eighth output text; Wherein, the eighth output text includes one or more fourth relation triples; each of the fourth relation triples is composed of a head entity text, an entity relation text and a tail entity text; Step 804, perform a round of traversal on all the fourth relationship triples of the eighth output text; and in this round of traversal, record the currently traversed fourth relationship triple as the corresponding current prediction triple; and record the second label relationship triple in the current label relationship triple sequence, whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relationship triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding fifth prediction-label pair by the current prediction triple and the corresponding paired label triple; Step 805, after traversing all the fourth relationship triples of the eighth output text, identify whether there are any second label relationship triples that have not been marked as the used triples in the current label relationship triple sequence; if there are any second label relationship triples that have not been marked as the used triples, use each of the second label relationship triples that have not been marked as the used triples as the corresponding remaining label triples, and use the invalid relationship triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding fifth prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples; Step 806, input all the fifth prediction-label pairs obtained into the first model loss function; and based on the preset third model optimizer, perform a round of parameter fine-tuning on the first functional parameter subset in the direction of minimizing the first model loss function; and at the end of this round of fine-tuning, identify whether the current training record is the last second data record of the third training set; if so, go to step 807; if not, extract the next second data record of the third training set as the new current training record and return to step 803; Wherein, the third model optimizer includes at least ADAM optimizer and AdamW optimizer; Step 807, taking the first second data record of the third evaluation set as the corresponding current evaluation record; Step 808, bringing the second training text of the current evaluation record into the first training instruction template to assemble the large model input text to obtain the corresponding ninth input text; and recording the second label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and inputting the ninth input text into the first model for processing to obtain the corresponding ninth output text; The ninth output text includes one or more fifth relation triples; each of the fifth relation triples is composed of a head entity text, an entity relation text and a tail entity text; Step 809, perform a round of traversal on all the fifth relation triples of the ninth output text; and in this round of traversal, record the fifth relation triple currently traversed as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence, whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding sixth prediction-label pair by the current prediction triple and the corresponding paired label triple; Step 810, after the traversal of all the fifth relationship triples of the ninth output text is completed, whether there are any second label relationship triples that have not been marked as the used triples in the current label relationship triple sequence are identified; if there are any second label relationship triples that have not been marked as the used triples, then each of the second label relationship triples that have not been marked as the used triples is used as the corresponding remaining label triple, and the invalid relationship triple is used as the paired prediction triple corresponding to each of the remaining label triples, and each of the remaining label triples and the corresponding paired prediction triple forms a corresponding sixth prediction-label pair; Step 811, identifying whether the current evaluation record is the last second data record of the third evaluation set; if so, going to step 812; if not, extracting the next second data record of the third evaluation set as the new current evaluation record and returning to step 808; Step 812, calculating the multiple evaluation parameters of the first model according to all the obtained sixth prediction-labels to obtain the corresponding third evaluation parameter set; and identifying whether the evaluation parameter values ​​of all the third evaluation parameters of the third evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one of the third evaluation parameters in the third evaluation parameter set does not satisfy the corresponding first parameter value range, returning to step 802 to continue training; if the evaluation parameter values ​​of all the third evaluation parameters satisfy their respective corresponding first parameter value ranges, stopping training and confirming that the fine-tuning of the relationship extraction function is completed; The third evaluation parameter set includes a plurality of the third evaluation parameters; the third evaluation parameters include the evaluation parameter type and the evaluation parameter value; and the evaluation parameter value of each of the third evaluation parameters has a corresponding first parameter value range.

9. A device for executing the method for performing relation extraction based on a large language model according to any one of claims 1 to 8, characterized in that: The device comprises: a model selection module, a model pre-training module, a model fine-tuning module, and a model application module; The model selection module is used to select a large language model as the corresponding first model; The model pre-training module is used to construct a benchmark label data set as the corresponding first data set through data collection, data cleaning and data annotation by manual or other machine models; and pre-train the relationship extraction function of the first model based on the first data set; and after the relationship extraction function pre-training is completed, the relationship verification function of the first model is pre-trained based on the first data set; The model fine-tuning module is used to construct an enhanced data set based on the text generation function of the first model as the corresponding second data set after the pre-training of the relationship verification function is completed; and to clean the second data set based on the relationship verification function of the first model; and to fine-tune the relationship extraction function of the first model based on the cleaned second data set; The model application module is used to, after the fine-tuning is completed, bring the first query text, the first restricted relationship sequence and the first output format input by the user into a preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and input the first input text into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user.

10. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 8; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images