A processing method and device for relation extraction based on large language model

Abstract

An embodiment of the present invention relates to a processing method and device for performing relationship extraction based on a large language model. The method includes: selecting a large language model as a first model; constructing a first data set through data collection / cleaning / labeling methods of manual / machine models to pre-train the relationship extraction / verification functions of the first model; after the pre-training, constructing a second data set based on the text generation function of the first model, and performing data cleaning on the second data set based on the relationship verification function of the first model, and fine-tuning the relationship extraction function of the first model based on the cleaned second data set; after the fine-tuning, bringing the user-input query text, limited relationship sequence, and output format into a first instruction template to assemble the large model input text, and inputting the assembled text into the first model to perform relationship triple recognition and screening processing to obtain output text and feedback to the user. The present invention can enhance model generalization and improve prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a processing method and device for performing relationship extraction based on a large language model. Background Art

[0002] With the advancement of artificial intelligence (AI), large language models (LLMs) have demonstrated impressive performance in natural language processing (NLP) tasks. By learning from massive amounts of text data, LLMs can understand and generate natural language text and are widely used in tasks such as text generation, translation, and question-answering systems. However, LLMs still face challenges in relation extraction (RE). Relation extraction, in this context, involves identifying relationships between entities in input text and outputting corresponding recognition results, such as a relation triple: head entity + entity relationship / association relationship + tail entity. Conventionally, training the relation extraction function of a large language model is mostly done through supervised training, which requires a large labeled dataset. Currently, preparing this labeled dataset often relies on manual data collection and annotation. This conventional approach, influenced by human factors, inevitably leads to issues such as insufficient data volume and limited scope. These challenges, in turn, pose challenges for the model, such as insufficient training, poor generalization, poor training quality, and low prediction accuracy. Summary of the Invention

[0003] The purpose of the present invention is to address the shortcomings of the prior art and provide a processing method, device, electronic device and computer-readable storage medium for relation extraction based on a large language model. The present invention selects a type of large language model as the corresponding first model; and constructs a benchmark label data set as the corresponding first data set through data collection, data cleaning and data annotation by manual or other machine models; and pre-trains the relation extraction function and relation verification function of the first model based on the first data set; and after the pre-training, constructs an enhanced data set as the corresponding second data set based on the text generation function of the first model; and cleans the second data set based on the relation verification function of the first model; and fine-tunes the relation extraction function of the first model based on the cleaned second data set; after the fine-tuning, the first query text, the first limited relation sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user. The present invention utilizes the text generation function of the large language model itself to add a method for constructing an enhanced dataset on top of the processing mechanism of the conventional labeled dataset, and improves the data accuracy of the enhanced dataset by training the relationship verification function of the large language model. In this way, the enhanced dataset with a large amount of data can be used to improve the training adequacy and training quality of the model, thereby achieving the purpose of enhancing the generalization and improving the model prediction accuracy. In addition, the present invention also uses the relationship extraction function of the large language model as the core to provide users with a processing mechanism for identifying and screening relationship triples of specified query text through the first instruction template, which can not only simplify the user operation steps but also improve the user experience.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present invention provides a method for performing relationship extraction based on a large language model, the method comprising:

[0005] Select a large language model as the corresponding first model;

[0006] A benchmark label dataset is constructed by data collection, data cleaning, and data labeling by manual or other machine models, recorded as the corresponding first dataset; and a relationship extraction function of the first model is pre-trained based on the first dataset; and after the relationship extraction function pre-training is completed, a relationship verification function of the first model is pre-trained based on the first dataset;

[0007] After the pre-training of the relationship verification function is completed, an enhanced dataset is constructed based on the text generation function of the first model and recorded as the corresponding second dataset; the second dataset is cleaned based on the relationship verification function of the first model; and the relationship extraction function of the first model is fine-tuned based on the cleaned second dataset;

[0008] After fine-tuning is completed, the first query text, the first restricted relationship sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user.

[0009] Preferably, the first model includes at least a GPT series model, a GPT-RE model, a BERT model, a UIE series model, a QA4RE model, and a Wenxin model;

[0010] The first data set includes a plurality of first data records; the first data record includes a first training text and a first label relationship triple sequence; the first label relationship triple sequence includes one or more first label relationship triples; the first label relationship triple consists of a head entity text, an entity relationship text, and a tail entity text;

[0011] The second data set includes multiple second data records; the second data records include a second label relationship triple sequence and a second training text; the second label relationship triple sequence includes one or more second label relationship triples; the second label relationship triple consists of a head entity text, an entity relationship text and a tail entity text;

[0012] The first restricted relationship sequence consists of one or more first entity relationship texts;

[0013] The first output format includes at least an array sequence format with a head entity-entity relationship-tail entity field, a list format with a head entity-entity relationship-tail entity field, and a specified script format with a head entity-entity relationship-tail entity field; the specified script format includes at least a BAT script, an HTM script, a JSON script, and a Python script;

[0014] The first instruction template is a formatted text template, which is used to use the first query text, the first restricted relationship sequence and the first output format as the corresponding current query text, the current restricted relationship sequence and the current output format, and to inform the large language model through a preset narrative text to extract relationship triples from the current query text, filter all extracted relationship triples based on the current restricted relationship sequence, and assemble and output the output text of the triple relationships that meet the current restricted relationship sequence according to the current output format.

[0015] Preferably, the reference label dataset constructed by manual or other machine model data collection, data cleaning and data labeling is recorded as the corresponding first dataset, specifically including:

[0016] Performing big data collection on text information in multiple fields by manual or other machine model data collection methods to obtain corresponding multiple first collected texts;

[0017] and performing grammatical and content compliance checks on all the obtained first collected texts by means of manual or other machine model data verification, and deleting the first collected texts that fail the grammatical check or the content compliance check; and performing duplicate text elimination processing on the remaining first collected texts by means of manual or other machine model text deduplication; and performing text similarity calculation on every two first collected texts after text deduplication by means of manual or other machine model text comparison, and deleting any one of the two first collected texts whose text similarity exceeds a preset similarity threshold;

[0018] And perform a round of traversal on all the remaining first collected texts; and in this round of traversal, take the first collected text currently traversed as the corresponding current collected text; and identify whether the current collected text contains extractable relationship triples through the relationship triple recognition method of manual or other machine models; if there are no extractable relationship triples in the current collected text, delete the current collected text; if the current collected text contains extractable relationship triples, take the current collected text as a corresponding first training text, and extract each relationship triple in the current collected text as a corresponding first label relationship triple, and form a corresponding first label relationship triple sequence from all the extracted first label relationship triples, and form a corresponding first data record from the obtained first training text and the first label relationship triple sequence; and at the end of this round of traversal, all the obtained first data records form the corresponding first data set.

[0019] Preferably, the pre-training of the relationship extraction function of the first model based on the first data set specifically includes:

[0020] Step 401: Split the first data set into two sub-data sets according to a preset first split ratio and record them as a corresponding first training set and a first evaluation set;

[0021] Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0022] Step 402: taking the first first data record of the first training set as the corresponding current training record;

[0023] Step 403: The first training text of the current training record is brought into a preset first training instruction template to perform large model input text assembly to obtain a corresponding first input text; and the first label relationship triple sequence of the current training record is recorded as the corresponding current label relationship triple sequence;

[0024] The first training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text, and to instruct the first model to perform relation triple recognition on the current text through a preset narrative text and output all recognized relation triples as model output text;

[0025] Step 404: input the first input text into the first model for processing to obtain a corresponding first output text;

[0026] The first output text includes one or more first relation triples; each of the first relation triples consists of a head entity text, an entity relation text, and a tail entity text;

[0027] Step 405: perform a round of traversal on all the first relation triples of the first output text; and in this round of traversal, record the first relation triple currently traversed as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use a preset invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding first prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0028] The invalid relationship triple is also composed of a head entity text, an entity relationship text, and a tail entity text, but the head entity text, the entity relationship text, and the tail entity text are all empty texts.

[0029] Step 406: After traversing all the first relation triples of the first output text, identify whether there are any first label relation triples in the current label relation triple sequence that have not been marked as the used triples; if there are any first label relation triples that have not been marked as the used triples, use each of the first label relation triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relation triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding first prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples;

[0030] Step 407: Record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset; input all the obtained first prediction-label pairs into a preset first model loss function; and perform a round of parameter optimization on the first function parameter subset based on a preset first model optimizer in a direction to minimize the first model loss function;

[0031] Among them, the first model loss function includes at least a cross entropy loss function, a logarithmic loss function, a mean square error loss function and a mean absolute error loss function; the first model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer and an AdamW optimizer;

[0032] Step 408: Identify whether the current training record is the last first data record in the first training set; if so, proceed to step 409; if not, extract the next first data record in the first training set as the new current training record and return to step 403;

[0033] Step 409: taking the first first data record of the first evaluation set as the corresponding current evaluation record;

[0034] Step 410: The first training text of the current evaluation record is brought into the first training instruction template to perform large model input text assembly to obtain a corresponding second input text; the first label relationship triple sequence of the current evaluation record is recorded as the corresponding current label relationship triple sequence; and the second input text is input into the first model for processing to obtain a corresponding second output text.

[0035] The second output text includes one or more second relation triples; each of the second relation triples consists of a head entity text, an entity relation text, and a tail entity text;

[0036] Step 411, perform a round of traversal on all the second relation triples of the second output text; and in this round of traversal, record the currently traversed second relation triple as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding second prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0037] Step 412: After traversing all the second relation triples of the second output text, identify whether there are any first label relation triples that have not been marked as the used triples in the current label relation triple sequence; if there are any first label relation triples that have not been marked as the used triples, use each of the first label relation triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relation triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding second prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples;

[0038] Step 413: Identify whether the current evaluation record is the last first data record in the first evaluation set; if so, proceed to step 414; if not, extract the next first data record in the first evaluation set as the new current evaluation record and return to step 410;

[0039] Step 414: Calculate the multiple evaluation parameters of the first model based on all the obtained second prediction labels to obtain a corresponding first evaluation parameter set; and identify whether the evaluation parameter values ​​of all first evaluation parameters in the first evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one first evaluation parameter in the first evaluation parameter set does not satisfy the corresponding first parameter value range, return to step 402 to continue training; if the evaluation parameter values ​​of all first evaluation parameters satisfy their respective corresponding first parameter value ranges, stop training and confirm that the relationship extraction function pre-training is completed;

[0040] Among them, the first evaluation parameter set includes multiple first evaluation parameters; the first evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate, F1 score and BLEU score; the evaluation parameter value of each first evaluation parameter has a preset first parameter value range corresponding to it.

[0041] Preferably, the pre-training of the relationship verification function of the first model based on the first data set specifically includes:

[0042] Step 51: dividing the first data set into two sub-data sets according to a preset second division ratio and recording them as a corresponding second training set and a second evaluation set;

[0043] Wherein, both the second training set and the second evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the second training set to the second evaluation set satisfies the second segmentation ratio;

[0044] Step 52: taking the first first data record of the second training set as the corresponding current training record;

[0045] Step 53: Bring the first training text and each of the first label relationship triples of the current training record into a preset second training instruction template to assemble a large model input text to obtain a corresponding third input text;

[0046] The second training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text and the first label relationship triple as the corresponding current relationship triple, and to inform the first model through a preset narrative text whether the association relationship between the head entity and the tail entity reflected by the current relationship triple in the current text is correct, and output the correct or incorrect recognition result as the model output text;

[0047] Step 54: input each of the third input texts corresponding to the current training record into the first model for processing to obtain a corresponding third output text;

[0048] Wherein, the third output text includes correct and wrong;

[0049] Step 55: Perform a round of traversal on all the third output texts corresponding to the current training record; and during this round of traversal, use the currently traversed third output text as the corresponding current output text; and recognize the current output text; if the current output text is correct, set the corresponding first prediction data to 1; if the current output text is incorrect, set the corresponding first prediction data to 0; and configure a corresponding first label data for each first prediction data, and set all the first label data to 1; and form a corresponding third prediction-label pair by each first prediction data and the corresponding first label data;

[0050] Step 56: Record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset, record the model parameter subset related to the relationship verification function as the corresponding second function parameter subset, record the intersection of the first and second function parameter subsets as the corresponding first parameter intersection, and record the other model parameters in the second function parameter subset except the first parameter intersection as the corresponding second function private parameter subset; input all the third prediction-label pairs corresponding to the current training record into a preset second model loss function; and perform a round of parameter optimization on the second function private parameter subset based on a preset second model optimizer in the direction of minimizing the second model loss function;

[0051] The second model loss function includes at least a mean square error loss function, a mean absolute error loss function, and a cross entropy loss function; the second model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer, and an AdamW optimizer;

[0052] Step 57: Identify whether the current training record is the last first data record in the second training set; if so, proceed to step 58; if not, extract the next first data record in the second training set as the new current training record and return to step 53;

[0053] Step 58, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the currently traversed first data record as the corresponding current evaluation record; and bring the first training text and each first label relationship triple of the current evaluation record into the second training instruction template to perform large model input text assembly to obtain a corresponding fourth input text; and input each of the fourth input texts corresponding to the current evaluation record into the first model for processing to obtain a corresponding fourth output text; and identify each of the fourth output texts corresponding to the current evaluation record, and if the current fourth output text is correct, set the corresponding second prediction data to 1; if the current fourth output text is wrong, set the corresponding second prediction data to 0; and configure a corresponding second label data for each second prediction data, and all the second label data are set to 1; and each second prediction data and the corresponding second label data form a corresponding fourth prediction-label pair;

[0054] Step 59: After traversing all the first data records of the second evaluation set, multiple evaluation parameters of the first model are calculated based on all the fourth prediction-labels obtained to obtain a corresponding second evaluation parameter set; and whether the evaluation parameter values ​​of all second evaluation parameters in the second evaluation parameter set satisfy the corresponding second parameter value range is identified; if the evaluation parameter value of at least one second evaluation parameter in the second evaluation parameter set does not satisfy the corresponding second parameter value range, return to step 52 to continue training; if the evaluation parameter values ​​of all second evaluation parameters satisfy their respective corresponding second parameter value ranges, stop training and confirm that the relationship verification function pre-training is completed;

[0055] Among them, the second evaluation parameter set includes multiple second evaluation parameters; the second evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate and F1 score; the evaluation parameter value of each second evaluation parameter has a preset second parameter value range corresponding to it.

[0056] Preferably, the enhanced dataset constructed based on the text generation function of the first model is recorded as the corresponding second dataset, which specifically includes:

[0057] Step 61, initialize the first record counter to 1;

[0058] Step 62: In the preset integer range [1,n max ] and perform a random rounding to get the corresponding random number N;

[0059] Among them, the maximum value of the range n max , the random number N is a positive integer, 1≤N≤n max ;

[0060] Step 63: Substitute the random number N into the preset third training instruction template to assemble the large model input text to obtain a corresponding fifth input text; and input the fifth input text into the first model for processing to obtain a corresponding fifth output text;

[0061] The third training instruction template is a formatted text template, which is used to use the random number N as the corresponding current total number of triples x, and to instruct the first model through a preset narrative text to automatically generate x relationship triples in multiple fields based on all knowledge fields covered by the model itself, and output the obtained x relationship triples as model output text; the fifth output text includes N third relationship triples;

[0062] Step 64: The N third relationship triples obtained this time are brought into the preset fourth training instruction template to assemble the large model input text to obtain a corresponding sixth input text; and the sixth input text is input into the first model for processing to obtain a corresponding sixth output text;

[0063] The fourth training instruction template is a formatted text template, which is used to use the N third relationship triples as the corresponding current triple sequence, and to instruct the first model through a preset narrative text to automatically generate a creative text based on the current triple sequence as the model output text, and requires that the creative text content must contain the entity and relationship original text of the current triple sequence;

[0064] Step 65: Each of the third relation triples obtained this time is used as a corresponding second label relation triple; the N second label relation triples obtained this time are used to form a corresponding second label relation triple sequence; the sixth output text obtained this time is used as a corresponding second training text; and the second label relation triple sequence obtained this time and the second training text are used to form a corresponding second data record;

[0065] Step 66, adding 1 to the first record counter; and identifying whether the first record counter after adding 1 is greater than the preset maximum total number of records; if not, returning to step 62; if so, all the obtained second data records form the corresponding second data set.

[0066] Preferably, the data cleaning of the second data set based on the relationship verification function of the first model specifically includes:

[0067] A round of traversal is performed on all the second data records of the second data set; and in this round of traversal, the second data record currently traversed is used as the corresponding current data record; and the second training text and each second label relationship triple of the current data record are brought into the second training instruction template to assemble the large model input text to obtain a corresponding seventh input text; and each of the seventh input texts corresponding to the current data record is input into the first model for processing to obtain the corresponding seventh output text; and whether all the seventh output texts corresponding to the current data record are correct are identified; if not, the current data record is deleted; wherein, the seventh output text includes correct and incorrect.

[0068] Preferably, fine-tuning the relationship extraction function of the first model based on the cleaned second data set specifically includes:

[0069] Step 801: Split the second data set into two sub-data sets according to a preset third split ratio and record them as a corresponding third training set and a third evaluation set;

[0070] wherein both the third training set and the third evaluation set are composed of a plurality of second data records; and a ratio of the total number of records in the third training set to the total number of records in the third evaluation set satisfies the third segmentation ratio;

[0071] Step 802: taking the first second data record of the third training set as the corresponding current training record;

[0072] Step 803: The second training text of the current training record is brought into the first training instruction template for large model input text assembly to obtain a corresponding seventh input text; the second label relationship triple sequence of the current training record is recorded as the corresponding current label relationship triple sequence; and the eighth input text is input into the first model for processing to obtain a corresponding eighth output text.

[0073] The eighth output text includes one or more fourth relation triples; each of the fourth relation triples consists of a head entity text, an entity relation text, and a tail entity text;

[0074] Step 804: perform a round of traversal on all the fourth relation triples of the eighth output text; and in this round of traversal, record the currently traversed fourth relation triple as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding fifth prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0075] Step 805: After traversing all the fourth relationship triples of the eighth output text, identify whether there are any second label relationship triples that have not been marked as the used triples in the current label relationship triple sequence; if there are any second label relationship triples that have not been marked as the used triples, use each of the second label relationship triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relationship triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding fifth prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples;

[0076] In step 806, all the obtained fifth prediction-label pairs are input into the first model loss function; and based on the preset third model optimizer, a round of parameter fine-tuning is performed on the first functional parameter subset in a direction in which the first model loss function reaches a minimum value; and at the end of this round of fine-tuning, whether the current training record is the last second data record in the third training set is identified; if so, the process proceeds to step 807; if not, the next second data record in the third training set is extracted as the new current training record and the process returns to step 803;

[0077] Wherein, the third model optimizer includes at least ADAM optimizer and AdamW optimizer;

[0078] Step 807: taking the first second data record of the third evaluation set as the corresponding current evaluation record;

[0079] Step 808: Bring the second training text of the current evaluation record into the first training instruction template for large model input text assembly to obtain a corresponding ninth input text; record the second label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and input the ninth input text into the first model for processing to obtain a corresponding ninth output text.

[0080] The ninth output text includes one or more fifth relation triples; each of the fifth relation triples consists of a head entity text, an entity relation text, and a tail entity text;

[0081] Step 809: perform a round of traversal on all the fifth relation triples of the ninth output text; and in this round of traversal, record the currently traversed fifth relation triple as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding sixth prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0082] Step 810: After traversing all the fifth relationship triples of the ninth output text, identify whether there are any second label relationship triples in the current label relationship triple sequence that have not been marked as the used triples; if there are any second label relationship triples that have not been marked as the used triples, use each of the second label relationship triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relationship triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding sixth prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples;

[0083] Step 811: Identify whether the current evaluation record is the last second data record in the third evaluation set; if so, proceed to step 812; if not, extract the next second data record in the third evaluation set as the new current evaluation record and return to step 808;

[0084] Step 812: Calculate the multiple evaluation parameters of the first model based on all the obtained sixth prediction labels to obtain a corresponding third evaluation parameter set; and identify whether the evaluation parameter values ​​of all third evaluation parameters in the third evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one third evaluation parameter in the third evaluation parameter set does not satisfy the corresponding first parameter value range, return to step 802 to continue training; if the evaluation parameter values ​​of all third evaluation parameters satisfy their respective corresponding first parameter value ranges, stop training and confirm that the fine-tuning of the relationship extraction function is completed;

[0085] The third evaluation parameter set includes a plurality of third evaluation parameters; the third evaluation parameters include the evaluation parameter type and the evaluation parameter value; and the evaluation parameter value of each third evaluation parameter has a corresponding first parameter value range.

[0086] A second aspect of an embodiment of the present invention provides a device for implementing the processing method for relation extraction based on a large language model described in the first aspect, the device comprising: a model selection module, a model pre-training module, a model fine-tuning module, and a model application module;

[0087] The model selection module is used to select a large language model as the corresponding first model;

[0088] The model pre-training module is used to construct a benchmark label data set as the corresponding first data set through data collection, data cleaning and data labeling by manual or other machine models; and pre-train the relationship extraction function of the first model based on the first data set; and after the relationship extraction function pre-training is completed, pre-train the relationship verification function of the first model based on the first data set;

[0089] The model fine-tuning module is used to construct an enhanced data set based on the text generation function of the first model as a corresponding second data set after the pre-training of the relationship verification function is completed; and to clean the second data set based on the relationship verification function of the first model; and to fine-tune the relationship extraction function of the first model based on the cleaned second data set;

[0090] The model application module is used to, after fine-tuning is completed, bring the first query text, the first restricted relationship sequence and the first output format input by the user into a preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and input the first input text into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user.

[0091] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0092] The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0093] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0094] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0095] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for relation extraction based on a large language model. As can be seen from the above content, the embodiment of the present invention selects a type of large language model as the corresponding first model; and constructs a benchmark label data set recorded as the corresponding first data set through data collection, data cleaning and data annotation of manual or other machine models; and pre-trains the relation extraction function and relation verification function of the first model based on the first data set; and after the pre-training, constructs an enhanced data set recorded as the corresponding second data set based on the text generation function of the first model; and cleans the second data set based on the relation verification function of the first model; and fine-tunes the relation extraction function of the first model based on the cleaned second data set; after the fine-tuning, the first query text, the first limited relation sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user. The embodiment of the present invention utilizes the text generation function of the large language model itself to add a method for constructing an enhanced dataset on top of the processing mechanism of a conventional labeled dataset, and improves the data accuracy of the enhanced dataset by training the relationship verification function of the large language model. The fine-tuning training of the model with the enhanced dataset improves the training adequacy and training quality of the model, enhances the generalization of the model, and improves the model prediction accuracy. In addition, the embodiment of the present invention also uses the relationship extraction function of the large language model as the core to provide users with a processing mechanism through the first instruction template that can identify and filter relationship triples of specified query text. This processing mechanism simplifies the user operation steps and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A schematic diagram of a processing method for performing relationship extraction based on a large language model provided in the first embodiment of the present invention;

[0097] Figure 2 A module structure diagram of a processing device for performing relationship extraction based on a large language model provided in the second embodiment of the present invention;

[0098] Figure 3 This is a structural diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0099] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely some, rather than all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0100] The first embodiment of the present invention provides a method for processing relation extraction based on a large language model, such as Figure 1 A schematic diagram of a method for performing relation extraction based on a large language model provided in the first embodiment of the present invention is shown. The method mainly includes the following steps:

[0101] Step 1: Select a large language model as the corresponding first model.

[0102] Here, the first model of the embodiment of the present invention includes at least large language models such as the GPT series model, the GPT-RE model, the BERT model, the UIE series model, the QA4RE model, and the Wenxin large model.

[0103] Step 2: construct a benchmark label dataset by manually or other machine model data collection, data cleaning, and data labeling, which is recorded as the corresponding first dataset; and pre-train the relationship extraction function of the first model based on the first dataset; and after the relationship extraction function pre-training is completed, pre-train the relationship verification function of the first model based on the first dataset;

[0104] Specifically comprising: step 21, constructing a benchmark label data set as the corresponding first data set through data collection, data cleaning and data labeling by manual or other machine models;

[0105] The first data set includes a plurality of first data records; the first data record includes a first training text and a first label relation triple sequence; the first label relation triple sequence includes one or more first label relation triples (RTE); the first label relation triple consists of a head entity (Head Entity / Subject) text, an entity relation (Relation) text, and a tail entity (Tail Entity / Object) text;

[0106] Specifically, step 211 includes collecting large amounts of text information from multiple fields by manual or other machine-based data collection methods to obtain corresponding first collected texts.

[0107] In step 212, all the first collected texts are subjected to grammatical and content compliance checks by manual or other machine model data verification methods, and first collected texts that fail the grammatical or content compliance checks are deleted; duplicate text removal is performed on the remaining first collected texts by manual or other machine model text deduplication methods; and text similarity is calculated for each two first collected texts after text deduplication by manual or other machine model text comparison methods, and any one of the two first collected texts whose text similarity exceeds a preset similarity threshold is deleted;

[0108] Here, the preset similarity threshold is a pre-set threshold parameter;

[0109] Step 213, and perform a round of traversal on all the remaining first collected texts; and in this round of traversal, the first collected text currently traversed is used as the corresponding current collected text; and the current collected text is identified by a manual or other machine model relationship triple recognition method to determine whether it contains extractable relationship triples; if there are no extractable relationship triples in the current collected text, the current collected text is deleted; if the current collected text contains extractable relationship triples, the current collected text is used as a corresponding first training text, and each relationship triple in the current collected text is extracted as a corresponding first label relationship triple, and all the extracted first label relationship triples are used to form a corresponding first label relationship triple sequence, and the obtained first training text and the first label relationship triple sequence are used to form a corresponding first data record; and at the end of this round of traversal, all the obtained first data records are used to form a corresponding first data set;

[0110] Step 22, pre-training the relationship extraction function of the first model based on the first data set;

[0111] Specifically comprising: step 22-1, dividing the first data set into two sub-data sets according to a preset first division ratio and recording them as a corresponding first training set and a first evaluation set;

[0112] The first split ratio is a preset ratio, for example, 8:2; the first training set and the first evaluation set both consist of a plurality of first data records; and the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first split ratio;

[0113] Step 22-2, taking the first first data record of the first training set as the corresponding current training record;

[0114] Step 22-3: The first training text of the current training record is brought into the preset first training instruction template to perform large model input text assembly to obtain the corresponding first input text; and the first label relationship triple sequence of the current training record is recorded as the corresponding current label relationship triple sequence;

[0115] The first training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text, and to instruct the first model to perform relation triple recognition on the current text through a preset narrative text and output all recognized relation triples as model output text;

[0116] Step 22-4, inputting the first input text into the first model for processing to obtain a corresponding first output text;

[0117] The first output text includes one or more first relation triples; each first relation triple is composed of a head entity text, an entity relation text, and a tail entity text;

[0118] Step 22-5, perform a round of traversal on all first relation triples of the first output text; and in this round of traversal, record the currently traversed first relation triple as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the preset invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding first prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0119] Among them, the invalid relation triple is also composed of the head entity text, entity relation text and tail entity text, but its head entity text, entity relation text and tail entity text are all empty text;

[0120] Here, the current step is actually to configure a paired label triple for each predicted first relation triple. The configuration method of this label triple is: query whether there is a matching first label relation triple (with the head entity as the alignment matching relationship) in the corresponding first label relation triple sequence. If there is, the matching first label relation triple is used as the label triple of the current first relation triple. If not, the label triple of the current first relation triple is set to a completely empty invalid relation triple.

[0121] Step 22-6: After traversing all first relation triples of the first output text, identify whether there are any first label relation triples in the current label relation triple sequence that have not been marked as used triples; if there are any first label relation triples that have not been marked as used triples, use each first label relation triple that has not been marked as a used triple as a corresponding remaining label triple, and use the invalid relation triple as a paired prediction triple corresponding to each remaining label triple, and form a corresponding first prediction-label pair from each remaining label triple and the corresponding paired prediction triple.

[0122] Here, the current step is actually to configure a completely empty invalid relation triple as a paired predicted triple for the unpredicted relation triple, that is, the first label relation triple in the first label relation triple sequence that is not marked as a used triple;

[0123] Step 22-7: Record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset; input all the obtained first prediction-label pairs into a preset first model loss function; and perform a round of parameter optimization on the first function parameter subset based on the preset first model optimizer in a direction to minimize the first model loss function;

[0124] Among them, the first model loss function includes at least a cross entropy loss function, a logarithmic loss function, a mean square error loss function, and a mean absolute error loss function; the first model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer, and an AdamW optimizer;

[0125] Step 22-8, identifying whether the current training record is the last first data record of the first training set; if so, going to step 22-9; if not, extracting the next first data record of the first training set as the new current training record and returning to step 22-3;

[0126] Step 22-9, taking the first first data record of the first evaluation set as the corresponding current evaluation record;

[0127] Step 22-10: Bring the first training text of the current evaluation record into the first training instruction template to assemble the large model input text to obtain the corresponding second input text; and record the first label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and input the second input text into the first model for processing to obtain the corresponding second output text;

[0128] The second output text includes one or more second relation triples; each second relation triple is composed of a head entity text, an entity relation text, and a tail entity text;

[0129] Step 22-11, perform a round of traversal on all second relation triples of the second output text; and in this round of traversal, record the currently traversed second relation triple as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding second prediction-label pair by the current prediction triple and the corresponding pairing label triple;

[0130] Step 22-12: After traversing all second relation triples of the second output text, identify whether there are any first label relation triples that have not been marked as used triples in the current label relation triple sequence; if there are any first label relation triples that have not been marked as used triples, use each first label relation triple that has not been marked as a used triple as a corresponding remaining label triple, and use the invalid relation triple as a paired prediction triple corresponding to each remaining label triple, and form a corresponding second prediction-label pair from each remaining label triple and the corresponding paired prediction triple.

[0131] Step 22-13, identifying whether the current evaluation record is the last first data record of the first evaluation set; if so, going to step 22-14; if not, extracting the next first data record of the first evaluation set as the new current evaluation record and returning to step 22-10;

[0132] Step 22-14, calculating the multiple evaluation parameters of the first model based on all the obtained second prediction-labels to obtain a corresponding first evaluation parameter set; and identifying whether the evaluation parameter values ​​of all first evaluation parameters in the first evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one first evaluation parameter in the first evaluation parameter set does not satisfy the corresponding first parameter value range, returning to step 22-2 to continue training; if the evaluation parameter values ​​of all first evaluation parameters satisfy their respective corresponding first parameter value ranges, stopping training and confirming that the relationship extraction function pre-training is completed;

[0133] The first evaluation parameter set includes a plurality of first evaluation parameters; the first evaluation parameters include an evaluation parameter type and an evaluation parameter value; the evaluation parameter type includes at least accuracy, precision, recall, F1 score, and BLEU score; and the evaluation parameter value of each first evaluation parameter has a preset first parameter value range corresponding thereto;

[0134] Step 23: After the pre-training of the relationship extraction function is completed, the relationship verification function of the first model is pre-trained based on the first data set;

[0135] Specifically comprising: step 231, dividing the first data set into two sub-data sets according to a preset second division ratio and recording them as corresponding second training sets and second evaluation sets;

[0136] The second split ratio is a preset ratio, for example, 8:2; the second training set and the second evaluation set are both composed of a plurality of first data records; the ratio of the total number of records in the second training set to the second evaluation set satisfies the second split ratio;

[0137] Step 232, taking the first first data record of the second training set as the corresponding current training record;

[0138] Step 233: The first training text of the current training record and each first label relationship triple are brought into a preset second training instruction template to assemble the large model input text to obtain a corresponding third input text;

[0139] The second training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text and the first label relationship triple as the corresponding current relationship triple, and to inform the first model through a preset narrative text whether the association relationship between the head entity and the tail entity reflected by the current relationship triple in the current text is correct, and output the specific correct or incorrect recognition result as the model output text;

[0140] Step 234: input each third input text corresponding to the current training record into the first model for processing to obtain a corresponding third output text;

[0141] Wherein, the third output text includes correct and wrong;

[0142] Step 235: Perform a round of traversal on all third output texts corresponding to the current training record; during this round of traversal, use the currently traversed third output text as the corresponding current output text; and recognize the current output text; if the current output text is correct, set the corresponding first prediction data to 1; if the current output text is incorrect, set the corresponding first prediction data to 0; and configure a corresponding first label data for each first prediction data, and set all first label data to 1; and form a corresponding third prediction-label pair from each first prediction data and the corresponding first label data;

[0143] Step 236: Record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset, record the model parameter subset related to the relationship verification function as the corresponding second function parameter subset, record the intersection of the first and second function parameter subsets as the corresponding first parameter intersection, and record the other model parameters in the second function parameter subset except the first parameter intersection as the corresponding second function private parameter subset; input all third prediction-label pairs corresponding to the current training record into the preset second model loss function; and perform a round of parameter optimization on the second function private parameter subset based on the preset second model optimizer in the direction of minimizing the second model loss function;

[0144] Among them, the second model loss function includes at least a mean square error loss function, a mean absolute error loss function and a cross entropy loss function; the second model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer and an AdamW optimizer;

[0145] Step 237: Identify whether the current training record is the last first data record of the second training set; if so, go to step 238; if not, extract the next first data record of the second training set as the new current training record and return to step 233;

[0146] Step 238, perform a round of traversal on all first data records of the second evaluation set; and in this round of traversal, use the currently traversed first data record as the corresponding current evaluation record; and bring the first training text and each first label relationship triple of the current evaluation record into the second training instruction template for large model input text assembly to obtain a corresponding fourth input text; and input each fourth input text corresponding to the current evaluation record into the first model for processing to obtain a corresponding fourth output text; and identify each fourth output text corresponding to the current evaluation record, and if the current fourth output text is correct, set the corresponding second prediction data to 1; if the current fourth output text is wrong, set the corresponding second prediction data to 0; and configure a corresponding second label data for each second prediction data, and set all second label data to 1; and form a corresponding fourth prediction-label pair by each second prediction data and the corresponding second label data;

[0147] Step 239: After traversing all first data records of the second evaluation set, multiple evaluation parameters of the first model are calculated based on all obtained fourth prediction labels to obtain a corresponding second evaluation parameter set; and whether the evaluation parameter values ​​of all second evaluation parameters in the second evaluation parameter set satisfy the corresponding second parameter value range is identified; if the evaluation parameter value of at least one second evaluation parameter in the second evaluation parameter set does not satisfy the corresponding second parameter value range, return to step 232 to continue training; if the evaluation parameter values ​​of all second evaluation parameters satisfy their respective corresponding second parameter value ranges, stop training and confirm that the relationship verification function pre-training is completed;

[0148] Among them, the second evaluation parameter set includes multiple second evaluation parameters; the second evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate and F1 score; the evaluation parameter value of each second evaluation parameter has a preset second parameter value range corresponding to it.

[0149] Step 3: After the pre-training of the relationship verification function is completed, an enhanced dataset is constructed based on the text generation function of the first model and recorded as the corresponding second dataset; the second dataset is cleaned based on the relationship verification function of the first model; and the relationship extraction function of the first model is fine-tuned based on the cleaned second dataset;

[0150] Specifically comprising: step 31, after the relationship verification function pre-training is completed, an enhanced data set is constructed based on the text generation function of the first model and recorded as the corresponding second data set;

[0151] The second data set includes a plurality of second data records; the second data records include a second label relationship triple sequence and a second training text; the second label relationship triple sequence includes one or more second label relationship triples; the second label relationship triple consists of a head entity text, an entity relationship text, and a tail entity text;

[0152] Specifically comprising: step 311, initializing a first record counter to 1;

[0153] Step 312: In the preset integer range [1,n max ] and perform a random rounding to get the corresponding random number N;

[0154] Among them, the maximum value of the range n max , the random number N is a positive integer, 1≤N≤n max ;

[0155] Step 313: Substitute the random number N into the preset third training instruction template to assemble the large model input text to obtain a corresponding fifth input text; and input the fifth input text into the first model for processing to obtain a corresponding fifth output text;

[0156] The third training instruction template is a formatted text template, which is used to use a random number N as the corresponding current total number of triples x, and to instruct the first model through a preset narrative text to automatically generate x relationship triples in multiple fields based on all knowledge fields covered by the model itself, and output the obtained x relationship triples as model output text; the fifth output text includes N third relationship triples;

[0157] Here, the third training instruction template of the embodiment of the present invention utilizes the knowledge domain scope of the first model itself to improve the domain width of the data;

[0158] Step 314: The N third relationship triples obtained this time are brought into the preset fourth training instruction template to assemble the large model input text to obtain a corresponding sixth input text; and the sixth input text is input into the first model for processing to obtain a corresponding sixth output text;

[0159] The fourth training instruction template is a formatted text template, which is used to take N third-relation triples as the corresponding current triple sequence, and instruct the first model through a preset narrative text to automatically generate a creative text based on the current triple sequence as the model output text output, and requires that the creative text content must contain the original entity and relationship text of the current triple sequence;

[0160] Here, the fourth training instruction template of the embodiment of the present invention utilizes the text generation capability of the first model itself to achieve the purpose of automatically generating enhanced data;

[0161] Step 315: Each third relation triple obtained this time is used as a corresponding second label relation triple; the N second label relation triples obtained this time are combined into a corresponding second label relation triple sequence; the sixth output text obtained this time is used as a corresponding second training text; and the second label relation triple sequence obtained this time and the second training text are combined into a corresponding second data record;

[0162] Step 316: increment the first record counter by 1; and determine whether the incremented first record counter is greater than a preset maximum total number of records; if not, return to step 312; if so, compose a corresponding second data set from all the obtained second data records;

[0163] Here, the maximum number of records is a pre-set large integer, which can be set based on the training sufficiency requirement. By treating the maximum number of records as a large integer, a large number of enhanced datasets with high data quality and spanning all model knowledge domains, i.e., the second dataset, can be obtained.

[0164] Step 32, performing data cleaning on the second data set based on the relationship verification function of the first model;

[0165] Specifically, the method includes: performing a round of traversal on all second data records of the second data set; and during this round of traversal, taking the currently traversed second data record as the corresponding current data record; bringing the second training text and each second label relationship triple of the current data record into the second training instruction template to perform large model input text assembly to obtain a corresponding seventh input text; and inputting each seventh input text corresponding to the current data record into the first model for processing to obtain a corresponding seventh output text; and determining whether all the seventh output texts corresponding to the current data record are correct; if not, deleting the current data record;

[0166] The seventh output text includes correct and incorrect;

[0167] Here, the embodiment of the present invention utilizes the relationship verification function of the first model to further improve the data accuracy of the second data set;

[0168] Step 33, fine-tuning the relationship extraction function of the first model based on the cleaned second data set;

[0169] Specifically comprising: step 33-1, dividing the second data set into two sub-data sets according to a preset third division ratio and recording them as a corresponding third training set and a third evaluation set;

[0170] The third split ratio is a preset ratio, for example, 8:2; the third training set and the third evaluation set both consist of a plurality of second data records; and the ratio of the total number of records in the third training set to the total number of records in the third evaluation set satisfies the third split ratio;

[0171] Step 33-2, taking the first second data record of the third training set as the corresponding current training record;

[0172] Step 33-3: The second training text of the current training record is brought into the first training instruction template for large model input text assembly to obtain the corresponding seventh input text; the second label relationship triple sequence of the current training record is recorded as the corresponding current label relationship triple sequence; and the eighth input text is input into the first model for processing to obtain the corresponding eighth output text;

[0173] The eighth output text includes one or more fourth relation triples; each fourth relation triple is composed of a head entity text, an entity relation text, and a tail entity text;

[0174] Step 33-4, perform a round of traversal on all fourth relation triples of the eighth output text; and in this round of traversal, record the currently traversed fourth relation triple as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding fifth prediction-label pair by the current prediction triple and the corresponding paired label triple;

[0175] Step 33-5: After traversing all fourth relationship triples of the eighth output text, identify whether there are any second label relationship triples in the current label relationship triple sequence that have not been marked as used triples; if there are any second label relationship triples that have not been marked as used triples, use each second label relationship triple that has not been marked as a used triple as a corresponding remaining label triple, and use the invalid relationship triple as a paired prediction triple corresponding to each remaining label triple, and form a corresponding fifth prediction-label pair from each remaining label triple and the corresponding paired prediction triple.

[0176] In step 33-6, all fifth prediction-label pairs obtained are input into the first model loss function; and based on the preset third model optimizer, a round of parameter fine-tuning is performed on the first functional parameter subset in a direction that minimizes the first model loss function; and at the end of this round of fine-tuning, whether the current training record is the last second data record of the third training set is identified; if so, the process proceeds to step 33-7; if not, the next second data record of the third training set is extracted as the new current training record and the process returns to step 33-3;

[0177] Wherein, the third model optimizer includes at least ADAM optimizer and AdamW optimizer;

[0178] Step 33-7, taking the first second data record of the third evaluation set as the corresponding current evaluation record;

[0179] Step 33-8: The second training text of the current evaluation record is brought into the first training instruction template to assemble the large model input text to obtain the corresponding ninth input text; the second label relationship triple sequence of the current evaluation record is recorded as the corresponding current label relationship triple sequence; and the ninth input text is input into the first model for processing to obtain the corresponding ninth output text;

[0180] The ninth output text includes one or more fifth relation triples; each fifth relation triple is composed of a head entity text, an entity relation text, and a tail entity text;

[0181] Step 33-9, perform a round of traversal on all fifth relation triples of the ninth output text; and in this round of traversal, record the currently traversed fifth relation triple as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the pairing label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the pairing label triple of the current prediction triple, and mark the current label triple as a used triple; and form a corresponding sixth prediction-label pair by the current prediction triple and the corresponding pairing label triple;

[0182] Step 33-10: After traversing all fifth relation triples of the ninth output text, identify whether there are any second label relation triples in the current label relation triple sequence that have not been marked as used triples; if there are any second label relation triples that have not been marked as used triples, use each second label relation triple that has not been marked as a used triple as a corresponding remaining label triple, and use the invalid relation triple as a paired prediction triple corresponding to each remaining label triple, and form a corresponding sixth prediction-label pair from each remaining label triple and the corresponding paired prediction triple.

[0183] Step 33-11, identifying whether the current evaluation record is the last second data record of the third evaluation set; if so, proceeding to step 33-12; if not, extracting the next second data record of the third evaluation set as the new current evaluation record and returning to step 33-8;

[0184] Step 33-12: Calculate the multiple evaluation parameters of the first model based on all the obtained sixth prediction labels to obtain a corresponding third evaluation parameter set; and identify whether the evaluation parameter values ​​of all third evaluation parameters in the third evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one third evaluation parameter in the third evaluation parameter set does not satisfy the corresponding first parameter value range, return to step 33-2 to continue training; if the evaluation parameter values ​​of all third evaluation parameters satisfy their respective corresponding first parameter value ranges, stop training and confirm that the fine-tuning of the relationship extraction function is completed;

[0185] Among them, the third evaluation parameter set includes multiple third evaluation parameters; the third evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate, F1 score and BLEU score; the evaluation parameter value of each third evaluation parameter has a corresponding first parameter value range.

[0186] Step 4: After fine-tuning is completed, the first query text, the first restricted relation sequence, and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; the first input text is then input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user;

[0187] Specifically, step 41 includes: after fine-tuning is completed, the first query text, the first restricted relationship sequence, and the first output format input by the user are brought into a preset first instruction template to assemble the large model input text to obtain the corresponding first input text;

[0188] Wherein, the first restricted relationship sequence consists of one or more first entity relationship texts;

[0189] The first output format includes at least an array sequence format with a head entity-entity relationship-tail entity field, a list format with a head entity-entity relationship-tail entity field, and a specified script format with a head entity-entity relationship-tail entity field; the specified script format includes at least a BAT script, an HTM script, a JSON script, and a Python script;

[0190] The first instruction template is a formatted text template, which is used to use the first query text, the first restricted relation sequence, and the first output format as the corresponding current query text, the current restricted relation sequence, and the current output format, and to instruct the large language model through a preset narrative text to extract relation triples from the current query text, filter all extracted relation triples based on the current restricted relation sequence, and assemble and output the output text of the triple relations that satisfy the current restricted relation sequence according to the current output format;

[0191] In step 42, the first input text is input into the first model for relation triple recognition and screening to obtain a corresponding first output text to be fed back to the user.

[0192] For example, the first model is known to be the Wenxinda model;

[0193] The text format of the first instruction template is:

[0194] "Current query text = <>,

[0195] Current qualified relation sequence = <>,

[0196] Current output format = <>,

[0197] Please extract relation triples from the current query text, filter all extracted relation triples based on the current restricted relation sequence, and assemble and output the triple relations that meet the current restricted relation sequence in the current output format.

[0198] The first query text "Narrative prose refers to prose that primarily records stories about people and is imbued with a strong lyrical atmosphere; its prominent features are the specific description and depiction of people and things, while also expressing the author's understanding and feelings" is introduced into the first instruction template. The first limiting relationship sequence "features" and the first output format "JSON script format with head entity-entity relationship-tail entity fields" are used to generate the first input text:

[0199] "Current query text = <Narrative prose refers to prose that primarily records stories about people and is imbued with a strong lyrical atmosphere; its prominent features are the specific description and depiction of people and things, while also expressing the author's understanding and feelings>,

[0200] Current qualified relationship sequence = <feature>,

[0201] Current output format = <JSON script format with header entity-entity relationship-tail entity fields>,

[0202] Please extract relation triples from the current query text, filter all extracted relation triples based on the current restricted relation sequence, and assemble and output the triple relations that meet the current restricted relation sequence in the current output format.

[0203] The JSON script obtained by inputting the first input text into the first model is:

[0204]

[0205] Figure 2 This is a module structure diagram of a processing device for relation extraction based on a large language model provided in the second embodiment of the present invention. The device is a terminal device or server that implements the aforementioned method embodiment, or can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: a model selection module 201, a model pre-training module 202, a model fine-tuning module 203 and a model application module 204.

[0206] The model selection module 201 is used to select a large language model as the corresponding first model.

[0207] The model pre-training module 202 is used to construct a benchmark label data set as the corresponding first data set through data collection, data cleaning and data annotation of manual or other machine models; and pre-train the relationship extraction function of the first model based on the first data set; and after the pre-training of the relationship extraction function is completed, pre-train the relationship verification function of the first model based on the first data set.

[0208] The model fine-tuning module 203 is used to construct an enhanced data set based on the text generation function of the first model as the corresponding second data set after the pre-training of the relationship verification function is completed; and to clean the second data set based on the relationship verification function of the first model; and to fine-tune the relationship extraction function of the first model based on the cleaned second data set.

[0209] The model application module 204 is used to, after the fine-tuning is completed, bring the first query text, the first restricted relationship sequence and the first output format input by the user into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and input the first input text into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user.

[0210] An embodiment of the present invention provides a processing device for relationship extraction based on a large language model, which can execute the method steps in the above method embodiment. Its implementation principles and technical effects are similar and will not be repeated here.

[0211] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by a processing element; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the model selection module can be a separately established processing element, or it can be integrated into a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above-mentioned determination module. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0212] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0213] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the above method embodiments are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) means. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, hard disk, tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0214] Figure 3 This is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present invention. The electronic device can be a terminal device or server that implements the method of the aforementioned embodiment, or it can be a terminal device or server that implements the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303's transceiver actions. Various instructions may be stored in the memory 302 for completing various processing functions and implementing the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0215] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0216] The above-mentioned processors can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0217] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the methods and processing procedures provided in the above embodiments.

[0218] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for relation extraction based on a large language model. As can be seen from the above content, the embodiment of the present invention selects a type of large language model as the corresponding first model; and constructs a benchmark label data set recorded as the corresponding first data set through data collection, data cleaning and data annotation of manual or other machine models; and pre-trains the relation extraction function and relation verification function of the first model based on the first data set; and after the pre-training, constructs an enhanced data set recorded as the corresponding second data set based on the text generation function of the first model; and cleans the second data set based on the relation verification function of the first model; and fine-tunes the relation extraction function of the first model based on the cleaned second data set; after the fine-tuning, the first query text, the first limited relation sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user. The embodiment of the present invention utilizes the text generation function of the large language model itself to add a method for constructing an enhanced dataset on top of the processing mechanism of a conventional labeled dataset, and improves the data accuracy of the enhanced dataset by training the relationship verification function of the large language model. The fine-tuning training of the model with the enhanced dataset improves the training adequacy and training quality of the model, enhances the generalization of the model, and improves the model prediction accuracy. In addition, the embodiment of the present invention also uses the relationship extraction function of the large language model as the core to provide users with a processing mechanism through the first instruction template that can identify and filter relationship triples of specified query text. This processing mechanism simplifies the user operation steps and improves the user experience.

[0219] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0220] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for relation extraction based on a large language model, characterized in that: The method comprises: Select a large language model as the corresponding first model; A benchmark label dataset is constructed by data collection, data cleaning, and data labeling by manual or other machine models, recorded as the corresponding first dataset; and a relationship extraction function of the first model is pre-trained based on the first dataset; and after the relationship extraction function pre-training is completed, a relationship verification function of the first model is pre-trained based on the first dataset; After the pre-training of the relationship verification function is completed, an enhanced dataset is constructed based on the text generation function of the first model and recorded as the corresponding second dataset; the second dataset is cleaned based on the relationship verification function of the first model; and the relationship extraction function of the first model is fine-tuned based on the cleaned second dataset; After fine-tuning is completed, the first query text, the first restricted relationship sequence and the first output format input by the user are brought into the preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and the first input text is input into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user.

2. The method for relation extraction based on a large language model according to claim 1, characterized in that: The first model includes at least the GPT series model, the GPT-RE model, the BERT model, the UIE series model, the QA4RE model, and the Wenxin model; The first data set includes a plurality of first data records; the first data record includes a first training text and a first label relationship triple sequence; the first label relationship triple sequence includes one or more first label relationship triples; The first label relationship triplet consists of a head entity text, an entity relationship text, and a tail entity text; The second data set includes a plurality of second data records; the second data records include a second label relation triple sequence and a second training text; The second label relationship triple sequence includes one or more second label relationship triples; The second label relationship triplet consists of a head entity text, an entity relationship text, and a tail entity text; The first restricted relationship sequence consists of one or more first entity relationship texts; The first output format includes at least an array sequence format with a head entity-entity relationship-tail entity field, a list format with a head entity-entity relationship-tail entity field, and a specified script format with a head entity-entity relationship-tail entity field; the specified script format includes at least a BAT script, an HTM script, a JSON script, and a Python script; The first instruction template is a formatted text template, which is used to use the first query text, the first restricted relationship sequence and the first output format as the corresponding current query text, the current restricted relationship sequence and the current output format, and to inform the large language model through a preset narrative text to extract relationship triples from the current query text, filter all extracted relationship triples based on the current restricted relationship sequence, and assemble and output the output text of the triple relationships that meet the current restricted relationship sequence according to the current output format.

3. The method for relation extraction based on a large language model according to claim 2, characterized in that: The benchmark label dataset constructed by manual or other machine model data collection, data cleaning and data labeling is recorded as the corresponding first dataset, specifically including: Performing big data collection on text information in multiple fields by manual or other machine model data collection methods to obtain corresponding multiple first collected texts; and performing grammatical and content compliance checks on all the obtained first collected texts by means of manual or other machine model data verification, and deleting the first collected texts that fail the grammatical check or the content compliance check; and performing duplicate text elimination processing on the remaining first collected texts by means of manual or other machine model text deduplication; and performing text similarity calculation on every two first collected texts after text deduplication by means of manual or other machine model text comparison, and deleting any one of the two first collected texts whose text similarity exceeds a preset similarity threshold; And perform a round of traversal on all the remaining first collected texts; and in this round of traversal, take the first collected text currently traversed as the corresponding current collected text; and identify whether the current collected text contains extractable relationship triples through the relationship triple recognition method of manual or other machine models; if there are no extractable relationship triples in the current collected text, delete the current collected text; if the current collected text contains extractable relationship triples, take the current collected text as a corresponding first training text, and extract each relationship triple in the current collected text as a corresponding first label relationship triple, and form a corresponding first label relationship triple sequence from all the extracted first label relationship triples, and form a corresponding first data record from the obtained first training text and the first label relationship triple sequence; and at the end of this round of traversal, all the obtained first data records form the corresponding first data set.

4. The method for relation extraction based on a large language model according to claim 2, characterized in that: The pre-training of the relationship extraction function of the first model based on the first data set specifically includes: Step 401: Split the first data set into two sub-data sets according to a preset first split ratio and record them as a corresponding first training set and a first evaluation set; Wherein, both the first training set and the first evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio; Step 402: taking the first first data record of the first training set as the corresponding current training record; Step 403: The first training text of the current training record is brought into a preset first training instruction template to perform large model input text assembly to obtain a corresponding first input text; and the first label relationship triple sequence of the current training record is recorded as the corresponding current label relationship triple sequence; The first training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text, and to instruct the first model to perform relation triple recognition on the current text through a preset narrative text and output all recognized relation triples as model output text; Step 404: input the first input text into the first model for processing to obtain a corresponding first output text; The first output text includes one or more first relation triples; each of the first relation triples consists of a head entity text, an entity relation text, and a tail entity text; Step 405: perform a round of traversal on all the first relation triples of the first output text; and in this round of traversal, record the first relation triple currently traversed as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use a preset invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding first prediction-label pair by the current prediction triple and the corresponding paired label triple; The invalid relationship triple is also composed of a head entity text, an entity relationship text, and a tail entity text, but the head entity text, the entity relationship text, and the tail entity text are all empty texts. Step 406: After traversing all the first relation triples of the first output text, identify whether there are any first label relation triples in the current label relation triple sequence that have not been marked as the used triples; if there are any first label relation triples that have not been marked as the used triples, use each of the first label relation triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relation triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding first prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples; Step 407: Record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset; input all the obtained first prediction-label pairs into a preset first model loss function; and perform a round of parameter optimization on the first function parameter subset based on a preset first model optimizer in a direction to minimize the first model loss function; Among them, the first model loss function includes at least a cross entropy loss function, a logarithmic loss function, a mean square error loss function and a mean absolute error loss function; the first model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer and an AdamW optimizer; Step 408: Identify whether the current training record is the last first data record in the first training set; if so, proceed to step 409; if not, extract the next first data record in the first training set as the new current training record and return to step 403; Step 409: taking the first first data record of the first evaluation set as the corresponding current evaluation record; Step 410: The first training text of the current evaluation record is brought into the first training instruction template to perform large model input text assembly to obtain a corresponding second input text; the first label relationship triple sequence of the current evaluation record is recorded as the corresponding current label relationship triple sequence; and the second input text is input into the first model for processing to obtain a corresponding second output text. The second output text includes one or more second relation triples; each of the second relation triples consists of a head entity text, an entity relation text, and a tail entity text; Step 411, perform a round of traversal on all the second relation triples of the second output text; and in this round of traversal, record the currently traversed second relation triple as the corresponding current prediction triple; and record the first label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding second prediction-label pair by the current prediction triple and the corresponding paired label triple; Step 412: After traversing all the second relation triples of the second output text, identify whether there are any first label relation triples that have not been marked as the used triples in the current label relation triple sequence; if there are any first label relation triples that have not been marked as the used triples, use each of the first label relation triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relation triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding second prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples; Step 413: Identify whether the current evaluation record is the last first data record in the first evaluation set; if so, proceed to step 414; if not, extract the next first data record in the first evaluation set as the new current evaluation record and return to step 410; Step 414: Calculate the multiple evaluation parameters of the first model based on all the obtained second prediction labels to obtain a corresponding first evaluation parameter set; and identify whether the evaluation parameter values ​​of all first evaluation parameters in the first evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one first evaluation parameter in the first evaluation parameter set does not satisfy the corresponding first parameter value range, return to step 402 to continue training; if the evaluation parameter values ​​of all first evaluation parameters satisfy their respective corresponding first parameter value ranges, stop training and confirm that the relationship extraction function pre-training is completed; Among them, the first evaluation parameter set includes multiple first evaluation parameters; the first evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate, F1 score and BLEU score; the evaluation parameter value of each first evaluation parameter has a preset first parameter value range corresponding to it.

5. The method for relation extraction based on a large language model according to claim 2, characterized in that: The pre-training of the relationship verification function of the first model based on the first data set specifically includes: Step 51: dividing the first data set into two sub-data sets according to a preset second division ratio and recording them as a corresponding second training set and a second evaluation set; Wherein, both the second training set and the second evaluation set are composed of a plurality of the first data records; the ratio of the total number of records in the second training set to the second evaluation set satisfies the second segmentation ratio; Step 52: taking the first first data record of the second training set as the corresponding current training record; Step 53: Bring the first training text and each of the first label relationship triples of the current training record into a preset second training instruction template to assemble a large model input text to obtain a corresponding third input text; The second training instruction template is a formatted text template, which is used to use the first training text as the corresponding current text and the first label relationship triple as the corresponding current relationship triple, and to inform the first model through a preset narrative text whether the association relationship between the head entity and the tail entity reflected by the current relationship triple in the current text is correct, and output the correct or incorrect recognition result as the model output text; Step 54: input each of the third input texts corresponding to the current training record into the first model for processing to obtain a corresponding third output text; Wherein, the third output text includes correct and wrong; Step 55: Perform a round of traversal on all the third output texts corresponding to the current training record; and during this round of traversal, use the currently traversed third output text as the corresponding current output text; and recognize the current output text; if the current output text is correct, set the corresponding first prediction data to 1; if the current output text is incorrect, set the corresponding first prediction data to 0; and configure a corresponding first label data for each first prediction data, and set all the first label data to 1; and form a corresponding third prediction-label pair by each first prediction data and the corresponding first label data; Step 56: Record the model parameter subset related to the relationship extraction function in the model parameter set of the first model as the corresponding first function parameter subset, record the model parameter subset related to the relationship verification function as the corresponding second function parameter subset, record the intersection of the first and second function parameter subsets as the corresponding first parameter intersection, and record the other model parameters in the second function parameter subset except the first parameter intersection as the corresponding second function private parameter subset; input all the third prediction-label pairs corresponding to the current training record into a preset second model loss function; and perform a round of parameter optimization on the second function private parameter subset based on a preset second model optimizer in the direction of minimizing the second model loss function; The second model loss function includes at least a mean square error loss function, a mean absolute error loss function, and a cross entropy loss function; the second model optimizer includes at least an SGD optimizer, an RMSprop optimizer, an ADAM optimizer, an Adafactor optimizer, and an AdamW optimizer; Step 57: Identify whether the current training record is the last first data record in the second training set; if so, proceed to step 58; if not, extract the next first data record in the second training set as the new current training record and return to step 53; Step 58, perform a round of traversal on all the first data records of the second evaluation set; and in this round of traversal, use the currently traversed first data record as the corresponding current evaluation record; and bring the first training text and each first label relationship triple of the current evaluation record into the second training instruction template to perform large model input text assembly to obtain a corresponding fourth input text; and input each of the fourth input texts corresponding to the current evaluation record into the first model for processing to obtain a corresponding fourth output text; and identify each of the fourth output texts corresponding to the current evaluation record, and if the current fourth output text is correct, set the corresponding second prediction data to 1; if the current fourth output text is wrong, set the corresponding second prediction data to 0; and configure a corresponding second label data for each second prediction data, and all the second label data are set to 1; and each second prediction data and the corresponding second label data form a corresponding fourth prediction-label pair; Step 59: After traversing all the first data records of the second evaluation set, multiple evaluation parameters of the first model are calculated based on all the fourth prediction-labels obtained to obtain a corresponding second evaluation parameter set; and whether the evaluation parameter values ​​of all second evaluation parameters in the second evaluation parameter set satisfy the corresponding second parameter value range is identified; if the evaluation parameter value of at least one second evaluation parameter in the second evaluation parameter set does not satisfy the corresponding second parameter value range, return to step 52 to continue training; if the evaluation parameter values ​​of all second evaluation parameters satisfy their respective corresponding second parameter value ranges, stop training and confirm that the relationship verification function pre-training is completed; Among them, the second evaluation parameter set includes multiple second evaluation parameters; the second evaluation parameters include evaluation parameter types and evaluation parameter values; the evaluation parameter types include at least accuracy, precision, recall rate and F1 score; the evaluation parameter value of each second evaluation parameter has a preset second parameter value range corresponding to it.

6. The method for relation extraction based on a large language model according to claim 2, characterized in that: The enhanced dataset constructed based on the text generation function of the first model is recorded as the corresponding second dataset, specifically including: Step 61, initialize the first record counter to 1; Step 62: In the preset integer range [1,n max ] and perform a random rounding to get the corresponding random number N; Among them, the maximum value of the range n max , the random number N is a positive integer, 1≤N≤n max ; Step 63: Substitute the random number N into the preset third training instruction template to assemble the large model input text to obtain a corresponding fifth input text; and input the fifth input text into the first model for processing to obtain a corresponding fifth output text; The third training instruction template is a formatted text template, which is used to use the random number N as the corresponding current total number of triples x, and to instruct the first model through a preset narrative text to automatically generate x relationship triples in multiple fields based on all knowledge fields covered by the model itself, and output the obtained x relationship triples as model output text; the fifth output text includes N third relationship triples; Step 64: The N third relationship triples obtained this time are brought into the preset fourth training instruction template to assemble the large model input text to obtain a corresponding sixth input text; and the sixth input text is input into the first model for processing to obtain a corresponding sixth output text; The fourth training instruction template is a formatted text template, which is used to use the N third relationship triples as the corresponding current triple sequence, and to instruct the first model through a preset narrative text to automatically generate a creative text based on the current triple sequence as the model output text, and requires that the creative text content must contain the entity and relationship original text of the current triple sequence; Step 65: Each of the third relation triples obtained this time is used as a corresponding second label relation triple; the N second label relation triples obtained this time are used to form a corresponding second label relation triple sequence; the sixth output text obtained this time is used as a corresponding second training text; and the second label relation triple sequence obtained this time and the second training text are used to form a corresponding second data record; Step 66, adding 1 to the first record counter; and identifying whether the first record counter after adding 1 is greater than the preset maximum total number of records; if not, returning to step 62; if so, all the obtained second data records form the corresponding second data set.

7. The method for relation extraction based on a large language model according to claim 5, characterized in that: The performing data cleaning on the second data set based on the relationship verification function of the first model specifically includes: A round of traversal is performed on all the second data records of the second data set; and in this round of traversal, the second data record currently traversed is used as the corresponding current data record; and the second training text and each second label relationship triple of the current data record are brought into the second training instruction template to assemble the large model input text to obtain a corresponding seventh input text; and each of the seventh input texts corresponding to the current data record is input into the first model for processing to obtain the corresponding seventh output text; and whether all the seventh output texts corresponding to the current data record are correct are identified; if not, the current data record is deleted; wherein, the seventh output text includes correct and incorrect.

8. The method for relation extraction based on a large language model according to claim 4, characterized in that: The fine-tuning of the relationship extraction function of the first model based on the cleaned second data set specifically includes: Step 801: Split the second data set into two sub-data sets according to a preset third split ratio and record them as a corresponding third training set and a third evaluation set; wherein both the third training set and the third evaluation set are composed of a plurality of second data records; and a ratio of the total number of records in the third training set to the total number of records in the third evaluation set satisfies the third segmentation ratio; Step 802: taking the first second data record of the third training set as the corresponding current training record; Step 803: The second training text of the current training record is brought into the first training instruction template for large model input text assembly to obtain a corresponding seventh input text; the second label relationship triple sequence of the current training record is recorded as the corresponding current label relationship triple sequence; and the seventh input text is input into the first model for processing to obtain a corresponding eighth output text. The eighth output text includes one or more fourth relation triples; each of the fourth relation triples consists of a head entity text, an entity relation text, and a tail entity text; Step 804: perform a round of traversal on all the fourth relation triples of the eighth output text; and in this round of traversal, record the currently traversed fourth relation triple as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding fifth prediction-label pair by the current prediction triple and the corresponding paired label triple; Step 805: After traversing all the fourth relationship triples of the eighth output text, identify whether there are any second label relationship triples that have not been marked as the used triples in the current label relationship triple sequence; if there are any second label relationship triples that have not been marked as the used triples, use each of the second label relationship triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relationship triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding fifth prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples; In step 806, all the obtained fifth prediction-label pairs are input into the first model loss function; and based on the preset third model optimizer, a round of parameter fine-tuning is performed on the first functional parameter subset in a direction in which the first model loss function reaches a minimum value; and at the end of this round of fine-tuning, whether the current training record is the last second data record in the third training set is identified; if so, the process proceeds to step 807; if not, the next second data record in the third training set is extracted as the new current training record and the process returns to step 803; Wherein, the third model optimizer includes at least ADAM optimizer and AdamW optimizer; Step 807: taking the first second data record of the third evaluation set as the corresponding current evaluation record; Step 808: Bring the second training text of the current evaluation record into the first training instruction template for large model input text assembly to obtain a corresponding ninth input text; record the second label relationship triple sequence of the current evaluation record as the corresponding current label relationship triple sequence; and input the ninth input text into the first model for processing to obtain a corresponding ninth output text. The ninth output text includes one or more fifth relation triples; each of the fifth relation triples consists of a head entity text, an entity relation text, and a tail entity text; Step 809: perform a round of traversal on all the fifth relation triples of the ninth output text; and in this round of traversal, record the currently traversed fifth relation triple as the corresponding current prediction triple; and record the second label relation triple in the current label relation triple sequence whose head entity text matches the head entity text of the current prediction triple and is not marked as a used triple as the corresponding current label triple; and identify whether the current label triple is empty; if the current label triple is empty, use the invalid relation triple as the paired label triple of the current prediction triple; if the current label triple is not empty, use the current label triple as the paired label triple of the current prediction triple, and mark the current label triple as the used triple; and form a corresponding sixth prediction-label pair by the current prediction triple and the corresponding paired label triple; Step 810: After traversing all the fifth relationship triples of the ninth output text, identify whether there are any second label relationship triples in the current label relationship triple sequence that have not been marked as the used triples; if there are any second label relationship triples that have not been marked as the used triples, use each of the second label relationship triples that have not been marked as the used triples as a corresponding remaining label triple, and use the invalid relationship triples as the paired prediction triples corresponding to each of the remaining label triples, and form a corresponding sixth prediction-label pair from each of the remaining label triples and the corresponding paired prediction triples; Step 811: Identify whether the current evaluation record is the last second data record in the third evaluation set; if so, proceed to step 812; if not, extract the next second data record in the third evaluation set as the new current evaluation record and return to step 808; Step 812: Calculate the multiple evaluation parameters of the first model based on all the obtained sixth prediction labels to obtain a corresponding third evaluation parameter set; and identify whether the evaluation parameter values ​​of all third evaluation parameters in the third evaluation parameter set satisfy the corresponding first parameter value range; if the evaluation parameter value of at least one third evaluation parameter in the third evaluation parameter set does not satisfy the corresponding first parameter value range, return to step 802 to continue training; if the evaluation parameter values ​​of all third evaluation parameters satisfy their respective corresponding first parameter value ranges, stop training and confirm that the fine-tuning of the relationship extraction function is completed; The third evaluation parameter set includes a plurality of third evaluation parameters; the third evaluation parameters include the evaluation parameter type and the evaluation parameter value; and the evaluation parameter value of each third evaluation parameter has a corresponding first parameter value range.

9. A device for executing the method for performing relationship extraction based on a large language model according to any one of claims 1 to 8, characterized in that: The device includes: a model selection module, a model pre-training module, a model fine-tuning module, and a model application module; The model selection module is used to select a large language model as the corresponding first model; The model pre-training module is used to construct a benchmark label data set as the corresponding first data set through data collection, data cleaning and data labeling by manual or other machine models; and pre-train the relationship extraction function of the first model based on the first data set; and after the relationship extraction function pre-training is completed, pre-train the relationship verification function of the first model based on the first data set; The model fine-tuning module is used to construct an enhanced data set based on the text generation function of the first model as a corresponding second data set after the pre-training of the relationship verification function is completed; and to clean the second data set based on the relationship verification function of the first model; and to fine-tune the relationship extraction function of the first model based on the cleaned second data set; The model application module is used to, after fine-tuning is completed, bring the first query text, the first restricted relationship sequence and the first output format input by the user into a preset first instruction template to assemble the large model input text to obtain the corresponding first input text; and input the first input text into the first model to perform relation triple recognition and screening processing to obtain the corresponding first output text and feedback to the user.

10. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 8; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

Images