Network message text element extraction method based on large language model integration, electronic device and computer-readable storage medium

Abstract

The invention relates to a network message text element extraction method, an electronic device and a computer-readable storage medium based on large language model integration, which belong to the field of text information element extraction. In order to solve the problem of integrating at element granularity, timely filtering and identifying erroneous elements, and improving the accuracy of generated elements, the technical key points are to use the confidence of any candidate element on any base model and the confidence of the termination token as the confidence of the proposition of the identification framework of the evidence reasoning rule, and generate evidence of any base model according to the confidence of the proposition of any base model in the identification framework; calculate the weighted confidence distribution of the evidence of any base model according to the evidence of any base model and the weight of any base model; according to the weighted confidence distribution of the evidence of any base model, fuse the evidence of all base models to obtain fused evidence, and use the element with the largest probability in the fused evidence as the integration result, so as to improve the accuracy of the extraction result.

Description

Technical Field

[0001] The invention belongs to the field of text information element extraction, and relates to a network message text element extraction method based on large language model integration, an electronic device and a computer-readable storage medium. Background Art

[0002] Online political inquiry refers to the process in which citizens make comments, suggestions or demands to government agencies through the Internet platform, and the government listens to the voice of the people and responds to their concerns through online channels. In the era of rapid development of network information technology, online political inquiry has become a new channel for communication and interaction between the government and the public. Local government portals, government microblogs and other online political inquiry platforms have been established one after another, greatly broadening the way for the public to participate in political life and building an efficient information bridge between the government and the public. The massive amount of online political inquiry data is like a mirror, truly reflecting the hot issues of social livelihood and the needs and expectations of the people. Processing and mining the key elements of political inquiry data can provide a scientific basis for government decision-making and effectively improve the efficiency of public services.

[0003] Information extraction is one of the key technologies in natural language processing, which aims to extract specific information from huge unstructured or semi-structured data. In the data of government messages, the key elements of information such as complaints, core demands, and solutions contained in citizen messages and department replies can help staff quickly understand the specific content and nature of the messages, facilitate the classification and location of problems, and provide a data basis for applications such as similar message recommendations. However, most of the large amount of government message data accumulated in the online government inquiry platform lack label information, making it difficult to directly apply traditional information extraction models that require large-scale data training for element extraction. On the other hand, unlike formal texts, government messages have colloquial features such as complex sentence components and casual expressions, which poses certain challenges to extracting element information from such informal texts. When there are problems such as missing components, redundancy, and irregular wording in the text, traditional information extraction methods based on sequence classification or pointer networks will perform poorly because they cannot identify valid information fragments; generative information extraction methods based on large language models show certain advantages on small sample data, but the lack of training corpus still limits the performance of the model. In addition, when the large language model generates a response sequence, the extraction effect of the preceding elements will affect the generation of the succeeding elements, and the resulting error transmission and error accumulation will also affect the final extraction result.

[0004] The integration strategy of existing language models mainly includes the following three aspects: First, provide the generation results of multiple base models to the large language model and let it generate the final result. Second, calculate the score of each base model generation result and vote to select the best result. Third, perform weighted integration of the conditional probability of the next word of each base model to generate the final result. For the first and second aspects, the existing research on large language model integration usually chooses to integrate the generation results of different models, but this method has some problems in integrating the granularity of generation results. First, if all the integrated models do not generate correct results, then the integration result cannot be correct. The obtained extraction result is only the best in the overall result, and each element is not necessarily the best. Secondly, for the feature extraction task, the model generation result contains multiple different elements and there is a dependency relationship between these elements. If the generation error of the previous element will affect the correct generation of the subsequent element. Therefore, directly integrating the generation results of different models cannot identify and correct the generated erroneous elements, thereby affecting the accuracy of the generation of the subsequent elements. Regarding the third aspect, this method is used in the field of text generation, and the conditional probability of the next word of each base model is weighted and integrated to generate the final result. This requires that the vocabulary of different base models is the same. Otherwise, if the vocabulary is different, for the same generated front word, different base models cannot all have the possibility of generating the same back word, and integration is impossible. Open source large language models usually have different vocabulary. The method of the third aspect mentioned above cannot perform model integration due to the different vocabulary of different base models. This usually means that a single large language model must be used to train multiple different base models, which makes it difficult to fully utilize the advantages of different large language models through integration. Therefore, this method has not been used in the field of text element extraction. Summary of the invention

[0005] In order to solve the problem of integrating at the element granularity, timely filtering and identifying erroneous elements, and improving the accuracy of generated elements, a network message text element extraction method based on large language model integration according to some embodiments of the present application is used to perform a task of extracting elements from a political message, wherein the elements include entity elements and / or relationship elements, and the extraction task includes an entity recognition task and / or a relationship recognition task;

[0006] The extraction method is applied to a large language model, the large language model includes at least two base models with different vocabularies, wherein the text describes the text of the element to be extracted, the instruction describes the type and output format of the element in the text of the element to be extracted, and the answer describes the extracted element output by the base model according to the instruction;

[0007] The network message text element extraction method based on large language model integration includes:

[0008] S100. Input a sample token sequence into the base model, wherein the sample token sequence includes an instruction, a text, and a token sequence of extracted elements;

[0009] The input sample token sequences of different base models are the same, but the vocabulary of different base models are different;

[0010] Wherein, if the sample token sequence does not have the extracted element, the token sequence of the extracted element is empty;

[0011] S200. Any base model outputs a token sequence of k candidate elements according to the input sample token sequence;

[0012] S300. Calculate the confidence of any candidate element in all candidate elements output by all base models on any base model, and calculate the confidence of the termination token;

[0013] S400. Assign weights to any base model;

[0014] S500. Taking the confidence of any candidate element on any base model and the confidence of the termination token as the confidence of the proposition of the identification framework of the evidence reasoning rule, and generating evidence of any base model according to the confidence of the proposition of any base model in the identification framework;

[0015] S600. Calculate the weighted confidence distribution of the evidence of any base model according to the evidence of any base model and the weight of any base model;

[0016] S700. According to the weighted confidence distribution of the evidence of any base model, the evidence of all base models is integrated to obtain integrated evidence, and the element with the highest probability in the integrated evidence is taken as the integration result.

[0017] If the integration result is a candidate element, the candidate element is taken as an extracted element and recorded in the answer, and steps S100-S700 are iteratively performed;

[0018] If the integration result is a termination token, the loop is stopped and the extracted elements recorded in the answer are output as the extracted elements output by the large language model.

[0019] In a second aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the first aspect and any possible technical solution of the first aspect.

[0020] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program. When the computer program runs on an electronic device, the electronic device executes the first aspect and any possible technical solution of the first aspect.

[0021] Beneficial effects: In the first aspect, the present invention is used in the field of text element extraction. In each round of iteration, a base model outputs k candidate elements in parallel through iteration. The present invention selects the best candidate elements (the number of candidate elements is k×m) output by different m base models as the pre-order extraction element through the method of evidence reasoning rule integration, and adds the pre-order extraction element to the input sample of the next round of iteration. The extraction reflects the dependency between elements. Moreover, the extraction elements generated by each round of iteration in the entire result are all selected as the best elements. Therefore, the overall result of the obtained extraction result is optimal, and each element is also optimal. The situation where the overall result is optimal but there are obviously wrong sub-elements is avoided. In addition, different base models of the present invention output k candidate elements respectively, and the elements come from the text. Therefore, the different vocabulary of each base model can provide candidate element diversity. Due to the selection of the best mechanism of evidence reasoning rule integration, the accuracy under the diversity can be further constrained. Therefore, the present invention can achieve the candidate diversity and result accuracy of integrated extraction.

[0022] On the second aspect, in the method for obtaining the token sequence of candidate elements of the present invention, the element name is iterated with multiple rounds of tokens, and in each round, the conditional probability of all token sequences that can be combined with the vocabulary token is calculated, and the best token sequences of this round are obtained from them, until a termination mark appears in a certain optimal token sequence of a certain round, indicating that one of the optimal token sequences (the optimal candidate element of the element name) appears, and after calculating the token sequence of the element name, the confidence of the token sequence of different element categories is generated, and the best one is selected to form an outputtable optimal candidate element, and k optimal candidate elements are output in multiple rounds of iterations according to the specified k. Therefore, the candidate elements used for integration of each base model in the present invention have themselves been selected through multiple rounds of iterations, and further combined with the above-mentioned method of selecting the best evidence reasoning rule for integration, the accuracy of the extraction results can be further improved.

[0023] In a further solution of the second aspect, since elements represented by simplified characters and elements represented by traditional characters usually have similar conditional probabilities, multiple candidate elements that are indistinguishable from traditional characters and simplified characters may appear in the result. The present invention skips all sequences containing traditional characters in the stage of selecting retained sequences to avoid multiple candidate elements that are indistinguishable from traditional characters and simplified characters appearing in the result.

[0024] In a further scheme on the second aspect, the output format of an entity element is "(element content, element category)", and the present invention only generates the element name of the entity candidate element when extracting the entity element. Since the eight entity element categories frequently appear as part of each entity element during the model training process. Therefore, when the model outputs the token sequence corresponding to the correct entity element category, the conditional probability of each token is very high. Since the length of each candidate sequence is exactly the same, if some sequences have shorter element content but longer element categories, their conditional probabilities may be higher than other sequences with longer element content. In order to reduce the impact of the element category length on the sequence score, the present invention stops generating a sequence after a sequence completes the generation of the element content part, that is, when the last token of the sequence is the token corresponding to the comma. Then, the confidence of each category generated after the sequence is calculated, and the category with the highest confidence is selected as the category of the candidate element.

[0025] In a further solution on the second aspect, when a large language model is used to generate text, it usually stops after the model outputs a termination token, but the present invention does not generate a termination token in the candidate element search method, but adds a termination token in each round of integration and calculates its confidence to determine the timing of terminating the generation of elements. Because the present invention uses three different large language models for integration, but the vocabulary of these three models is not the same, the same text sequence will be divided into different token sequences in different models. This means that when generating candidate elements after some elements have been generated, it is necessary to start generating after the type of the last element of the generated elements, so that other models can generate termination tokens in normal order. However, starting to generate after the type of the last element of the generated elements means that other content may be generated in the element type part of the last element of the generated elements, and additional tokens need to be generated to increase the calculation time when generating candidate elements. Therefore, the present invention does not generate a termination token in the candidate element search, and adding a termination token in the recognition framework can avoid these problems and reduce the complexity of the algorithm.

[0026] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Technology roadmap.

[0028] Figure 2 A framework for building a base model for feature extraction based on fine-tuning of a large language model.

[0029] Figure 3 Test results of the entity feature extraction task.

[0030] Figure 4 Test results of the relational element extraction task.

[0031] Figure 5 A framework for feature extraction methods based on evidence theory and ensemble learning.

[0032] Figure 6 The extraction results of different categories of entity features using the EG-ERRule method.

[0033] Figure 7 Model weight search iteration results based on NSGA-II algorithm.

[0034] Figure 8 The F1-0.8 index of the entity feature extraction task and the relationship feature extraction task of all individuals on the validation set.

[0035] Fig. 9 F1-0.8 index of entity feature extraction task and relationship feature extraction task under different model weights.

[0036] Fig.10 The overall architecture of the prototype system for recommending treatment measures based on political elements.

[0037] Fig.11 Feature extraction module interface. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present application in detail with reference to the accompanying drawings, examples of which are shown in the accompanying drawings. The present application provides a method and an electronic device. The method and the device are based on the same technical concept. Since the principles of solving problems by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0039] 1.1 Terminology

[0040] 1.1.1 Online government inquiry platform: As a new channel for communication between the government and the public in the information technology era, online government inquiry platforms make full use of the message data on online government inquiry platforms. Many scholars have conducted relevant research. At present, the mining of government inquiry data on online government inquiry platforms is mainly based on empirical research. At present, the research on message information mining mainly focuses on mining from a single aspect such as the complaint field or emotion of the message, and lacks research on the extraction of key elements such as the appeal and treatment measures of the message.

[0041] 1.1.2 Feature extraction: Feature extraction is a process of automatically identifying and extracting key information from unstructured or semi-structured data. This information usually includes structured elements such as entities, attributes, and relationships, in order to facilitate further data analysis, information retrieval, or knowledge base construction. Current research on feature extraction usually transforms feature extraction tasks in specific fields into information extraction tasks such as named entity recognition and relationship extraction, so as to utilize the rich methods and models in the field of information extraction to extract feature information in specific fields.

[0042] 1.1.3 Instruction Learning: Instruction Tuning, also known as instruction learning, is a method that uses a combination of natural language instructions and text to input into a pre-trained language model to perform specific natural language processing tasks. In this type of method, the instructions are usually a detailed description of the task, which may include task objectives, explanations of the proper nouns involved, output format requirements, precautions, etc. In traditional fine-tuning methods, models usually require a large number of input and output samples for training to learn the requirements of the task and output format. The instruction fine-tuning method inputs the instructions together with the text to be processed into a pre-trained language model, and requires the model to output according to the instructions.

[0043] 1.1.4 Ensemble Learning: Ensemble Learning is a method that combines multiple different prediction results from different models to obtain the final prediction result. Ensemble learning can combine multiple complementary models to improve the generalization ability of the integrated model, and is widely used in multiple natural language processing tasks such as intent recognition, relationship extraction, and sentiment classification.

[0044] (1) Using ensemble learning for information extraction: Current research on the use of ensemble learning in information extraction tasks usually assumes that the basic unit, the entity, appears in the form of raw text fragments. This makes it difficult to handle the problem that the fragments of political messages studied in the present invention, which have missing components, redundancy, and non-standard wording, cannot be directly used as political elements. Therefore, the present invention uses a large language model to perform element extraction tasks in the form of sequence generation, and designs a corresponding ensemble method to integrate multiple large language models.

[0045] (2) Use evidence theory to integrate model results: In integrated learning research, it is usually necessary to combine the results from different models. Commonly used integration strategies such as voting, averaging and weighted averaging are simply combinations of results and cannot effectively utilize the internal information of model results. The integration strategy based on evidence theory in the present invention can make full use of the distribution information of model results and can process model output results with uncertain information. The integration method based on evidence theory has more effective information fusion capabilities than traditional integration methods. Therefore, the text uses the evidence reasoning rule method to integrate the information of the large language model to select the optimal elements of the question message.

[0046] (3) Large language model integration: Existing research on large language model integration usually chooses to integrate the generation results of different models, but this method has some problems in integrating the generation result granularity. First, if all the integrated models do not generate correct results, then the integration results cannot be correct. Secondly, for feature extraction tasks, the model generation results contain multiple different features and there are dependencies between these features. If the generation error of the previous feature will affect the correct generation of the subsequent feature. Therefore, directly integrating the generation results of different models cannot identify and correct the generated erroneous features, thereby affecting the accuracy of the generation of the subsequent features.

[0047] 1.2 Research content and technical route: The technical route map of the present invention is as follows Figure 1As shown. The main research content of the present invention is to design a large language model for extracting political question elements, aiming to solve the problem of poor performance in extracting political question message elements under the condition of lack of labeled data. In order to improve the factor extraction effect, the present invention uses an ensemble learning method to combine the advantages of different large language models to improve the factor extraction effect. The premise of integrating sequence generation models to achieve efficient factor extraction is that at least one model can accurately generate political question elements. Therefore, the base model for integration needs to have a higher factor extraction capability. The first research content of the present invention is to use the instruction fine-tuning method to use other fields of information extraction data sets to improve the political question element extraction capability of the base model, and provide a model basis for subsequent integration. The second research content of the present invention is to design a factor extraction method framework based on ensemble learning, which integrates each element to timely discover and correct generation errors, improve factor extraction accuracy, and reduce the possibility of transmission errors.

[0048] Due to the lack of labeled data in the field of online political message element extraction, relevant studies have shown that increasing the number of instruction training tasks can improve the model's small-sample learning ability. Therefore, the present invention first constructs a Chinese information extraction instruction dataset. Since Chinese information extraction instruction datasets are currently scarce, existing Chinese information extraction instruction datasets such as KnowLM-IE are mainly labeled using remote supervision methods, and there is a certain amount of noise in the training samples. Therefore, the present invention collects multiple existing Chinese information extraction related datasets, and designs information extraction instructions to modify the datasets, and constructs a Chinese information extraction instruction dataset for pre-training of information extraction tasks. This dataset is then used to fine-tune the model.

[0049] Due to the huge parameter scale of the large language model, fine-tuning the model requires a huge amount of video memory and computing resources. Therefore, the present invention uses the LoRA method to fine-tune the basic large language model. Next, the present invention uses a web crawler method to crawl the political message data from the Hunan Online Political Inquiry Platform, and uses a manual annotation method to construct a political message element extraction instruction dataset, and uses this dataset to further fine-tune the model to transfer the general extraction capabilities learned by the model in other field datasets to the political message element extraction task, so as to achieve accurate extraction of message elements.

[0050] In view of the error propagation problem existing when a large language model generates a sequence, the present invention proposes a factor extraction method based on evidence theory and ensemble learning. The method first uses a search-based candidate factor search method to search for multiple candidate factors through multiple large language models, and uses the set of searched candidate factors as the identification framework for integration. Each large language model is used to calculate the conditional probability geometric mean of each candidate factor as the confidence of each candidate factor. In order to better integrate the information from multiple models, the present invention uses the NSGA-II algorithm to search for the weight of each model and uses the evidence reasoning rule method to fuse the candidate factor confidence information of multiple models to calculate the most likely next factor. This factor generation process is repeated continuously, so that the generation results of multiple large language models are integrated at the factor granularity, reducing the generation of erroneous factors and improving the accuracy of factor extraction.

[0051] Based on the results of the two parts of research work, the present invention designs and implements a prototype system for recommending treatment measures based on political inquiry messages, further verifies the effectiveness of the method proposed in the present invention, and takes the recommendation of treatment measures as an example to demonstrate the application scenarios and values ​​of the extracted political inquiry elements.

[0052] 2 Method for constructing a factor extraction base model based on fine-tuning of large language model instructions: The data in the field of online political inquiries lacks labeled data and is mostly informal text in spoken language. Traditional information extraction methods cannot adapt well to the requirements for extracting key elements of online political inquiries. In response to these problems, the present invention designs a method for constructing a factor extraction model based on fine-tuning of large language model instructions. The present invention first defines the key element categories of complaint-type online political inquiries. Then, information extraction instructions are designed, and corresponding complaint-type political inquiries instruction data sets are constructed. Then, a Chinese information extraction instruction data set is constructed based on several existing Chinese data sets, and the model is fine-tuned to train the general information extraction ability of the model. Subsequently, the model is further fine-tuned using the complaint-type political inquiries instruction data set, and the general information extraction ability learned by the model is transferred to the field of political inquiries element extraction. Finally, the method of the present invention is used on multiple open source large language models for training and tested on the complaint-type political inquiries instruction data set to verify the effectiveness of the method proposed by the present invention.

[0053] 2.1 Task definition of political message element extraction and framework of base model construction method

[0054] 2.1.1 Task definition for extracting elements from government message: Online government inquiry data includes two parts: citizen messages and responses from relevant departments. The citizen message part is the citizen’s description of the problem, and the relevant department’s response part is the response of the department responsible for handling the problem to the citizen, mainly including the department that handles the problem and its handling measures. Therefore, the present invention divides the key elements of government message into two categories: entity elements and relationship elements. Among them, entity elements are defined as the following eight categories: complaint problem, time of occurrence of complaint problem, location of occurrence of complaint problem, object of complaint, harm caused by complaint problem, appeal, handling department, and handling method. The meaning and examples of each type of entity element are shown in Table 2.1. Relationship elements are the corresponding relationship between handling departments and handling measures. Identifying such relationships helps to clarify the specific measures implemented by each specific handling department in the responses of relevant departments with multiple departments and multiple handling measures, thereby providing a data basis for applications such as a sharp reduction in handling measures for similar problems.

[0055] Table 2.1 Entity element categories of complaint-related messages

[0056]

[0057] At present, feature extraction mainly involves unified modeling of entity recognition, relationship extraction, and event extraction tasks. The use of unified information extraction task modeling is not only conducive to the model learning the general capabilities of different information extraction tasks, but also enables the use of more data sets in other fields for fine-tuning to improve the model extraction effect. The three common information extraction tasks of entity recognition, relationship extraction, and event extraction are usually divided into four subtasks: entity recognition, relationship recognition, event category recognition, and argument recognition, in order to reduce the complexity of each subtask and thus improve the extraction performance of the model. The goal of the entity recognition task is to generate a sequence:

[0058]

[0059] Among them, entity i Indicates entity value, entity_type i Indicates the entity type corresponding to the i-th entity.

[0060] The relation extraction task is divided into two subtasks: entity recognition and relation recognition. The entity recognition subtask recognizes the subject and object and their types in the input text; the relation recognition subtask generates a sequence based on this:

[0061] (relation1, object1, subject1), (realtion2, object2, subject2),..., (relation i , object i , subject i )

[0062] where relation i Indicates the relationship type, object i Indicates a specific subject, subject i Indicates a specific object.

[0063] The event extraction task is divided into two subtasks: event category recognition and argument recognition. The event category recognition subtask identifies the category of events present in the input. Its goal is to generate a sequence event1, event2, ..., event i , where event i Represents the event type; the argument recognition subtask identifies the arguments and their types of each event on this basis. The goal of the argument recognition subtask is to generate the sequence

[0064] (event1, (argument 11 , argument_type 11 ),...), (event2, (argument 21 , argument_type 21 ),…),…,(event i ,(argument i1 , argument_type i1 ), …)

[0065] Among them event i Indicates the type of event, argument i1 Indicates the first argument content of the event, argument_type i1 Indicates the argument type corresponding to the first argument of the event.

[0066] The extraction of political message elements in the present invention can be regarded as the above-mentioned sequence generation task, wherein the large language model of the present invention performs the task of extracting political message elements, the elements only involve entity elements and relationship elements, and the extraction task only involves entity recognition tasks and relationship recognition tasks. Entity element extraction is performed in the form of entity recognition tasks, and relationship element extraction is performed in the form of entity recognition and relationship recognition subtasks, wherein the relationship of the relationship recognition subtask has only one relationship "execution", whose subject is the processing department and the object is the processing measures executed by the processing department.

[0067] However, from the above, the present invention constructs a Chinese information extraction instruction dataset based on several existing Chinese datasets, and fine-tunes the model to train the general information extraction capability of the model. Subsequently, a complaint-type political inquiry message instruction dataset is used to further fine-tune the model, and the general information extraction capability learned by the model is transferred to the field of political inquiry message element extraction. Therefore, in the model training stage, the present invention involves several existing Chinese information extraction instruction datasets as training sets. Therefore, in the model training stage, the present invention involves event extraction tasks as well as event category recognition and argument recognition subtasks.

[0068] 2.1.2 Framework for building a base model for feature extraction based on fine-tuning of large language model instructions

[0069] Traditional entity extraction and relationship extraction methods usually require a large amount of labeled data, and sufficient model training can achieve good results. With the development of large language model technology and the improvement of large language model performance, instruction fine-tuning, as a technology that can effectively utilize large language models, acquire knowledge from training corpus during pre-training and enable the model to follow instructions to handle unknown tasks, is used to perform entity extraction and relationship extraction tasks in a few-sample environment. However, Chinese information extraction instruction data sets are currently scarce, and existing Chinese information extraction instruction data sets such as KnowLM-IE are mainly annotated using remote supervision methods, and there is a certain amount of noise in the training samples. In response to the above problems, the present invention proposes a construction of a factor extraction base model based on instruction fine-tuning.

[0070] The feature extraction model construction framework based on instruction fine-tuning is as follows Figure 2As shown. First, in order to better utilize the knowledge learned by the large language model during the pre-training process, the present invention represents entity extraction, relationship extraction and event extraction tasks as sequence generation tasks. Referring to the instruction design in the previous research on instruction fine-tuning, the present invention processes each information extraction task text into three parts: instructions, text and answers. The instruction part describes in detail the information to be extracted from the text to be extracted, the relevant requirements and the output format, etc., which are used to guide the model to correctly extract information from the text to be extracted and generate output in the correct format; the text refers to the sentence or paragraph that needs to be extracted. The text will be attached to the instruction and input into the large language model, so that the model outputs the corresponding extraction result according to the specific task requirements; the answer refers to the sentence generated by organizing the information extracted from the text according to the format requirements in the instruction, that is, the expected output sequence of the large language model after inputting the corresponding instruction and text. For the design details of the information extraction instruction, please refer to Section 2.2.1. Next, since there is currently no data set related to the extraction of elements from political messages, the research according to the present invention needs to construct a data set of instructions for extracting elements from political messages, manually annotate complaint-type political messages, and process them according to the instruction template to construct a data set of instructions for extracting elements from complaint-type political messages. In order to improve the element extraction performance of the model, it is necessary to construct data sets of information extraction instructions from other fields to fine-tune the model and train the general information extraction capabilities of the model. Therefore, the present invention processes the existing Chinese information extraction data set to construct a Chinese information extraction instruction data set. For details on the construction of the data set, see Section 2.2.2. Finally, the large language model is fine-tuned with instructions using the Chinese information extraction instruction data set and the complaint-type political element extraction instruction data set in turn. First, fine-tune the instructions on the Chinese information extraction instruction data set to improve the model's general information extraction capabilities and the ability to understand the extraction instructions, and then use the political element extraction instruction data set for further fine-tuning. Thereby achieving an element extraction model with better training effect when the data scale is limited. Due to the large number of model parameters, the traditional full-parameter fine-tuning method requires too much storage and computing resources. The text selection uses the LoRA method for fine-tuning, which reduces the storage and computing requirements during fine-tuning by reducing the number of parameters updated during fine-tuning. For details on model training, see Section 2.3.

[0071] 2.2 Two-stage information extraction instruction design and instruction dataset construction: The instruction fine-tuning method has been widely proven to enable the model to have good zero-sample and few-sample performance, but existing instruction fine-tuning research mainly focuses on English corpus and lacks Chinese information extraction instruction dataset. In order to improve the model's ability to extract elements from political messages, it is necessary to use a Chinese information extraction instruction dataset to pre-fine-tune the model before training with the political message data. Therefore, the present invention first designs two-stage information extraction instructions and corresponding instruction templates based on the instruction design method previously studied, and then constructs a complaint-type political message element extraction instruction dataset and a Chinese information extraction instruction dataset based on the instruction template.

[0072] 2.2.1 Two-stage information extraction instruction design: In order to better utilize the knowledge learned by the large language model during the pre-training process, the present invention represents entity extraction, relationship extraction and event extraction tasks as sequence generation tasks. Each label data of each specific extraction task will be processed into three parts: instructions, text and answers. The instruction part describes in detail the type of information to be extracted from the text to be extracted and the output format and other information, which is used to guide the model to correctly extract information from the text to be extracted and generate output in the correct format. The text part is the text from which information needs to be extracted, and the answer part is the result of processing the information in the text according to the output format requirements in the instruction. The present invention tests multiple instruction templates under zero-sample and few-sample conditions on the ChatGPT3.5 model, and selects the instruction template with the best effect. The instruction template used for the text is shown in Table 2.2. The parts in brackets such as "[Entity Type List]" need to be replaced with the actual categories that need to be extracted in the specific information extraction data. Taking the task of extracting entity elements from political messages in invention research as an example, "[Entity Type List]" needs to be replaced with "[complaint issue, time of occurrence of complaint issue, location of occurrence of complaint issue, object of complaint, harm caused by complaint issue, appeal, handling department, handling measures]".

[0073] Table 2.2 Instruction template example

[0074]

[0075]

[0076] Taking relation extraction as an example, the data sample S of the relation extraction task using two-stage instructions can be expressed as:

[0077] S={Instruction er ,Text,Answer er , Instruction re , Answer re}(2.1)

[0078] Instruction er is the instruction of the entity recognition subtask, Text is the text to be extracted, Answer er is the expected output of the entity recognition subtask, Instruction re Instructions for identifying subtasks for a relation, Answer re is the expected output of the relation recognition subtask. It should be noted that during training, the answer for the relation recognition subtask er Labels from data samples, Answers in actual extraction er is the output of the model in the entity recognition subtask.

[0079] From the above, compared with the model training of the government message instruction dataset, only two task types, entity recognition and relationship recognition, are used in the instruction template. It can be understood that this instruction template example can be used as a general instruction template to cover more task types, such as the model training of the Chinese information extraction instruction dataset.

[0080] 2.2.2 Instruction Dataset Construction

[0081] (1) Construction of a dataset of instructions for extracting elements from government messages: Although the government inquiry platform has accumulated a large amount of government message data, there is currently no label dataset suitable for the task of the present invention. Therefore, through operations such as data acquisition, processing, and labeling, the present invention first constructs a dataset of elements of complaint-type government message extraction that meets the task requirements. Specifically, the following will introduce the process of constructing a dataset of instructions for extracting elements from government messages, including data acquisition, preprocessing, labeling, and adding instructions.

[0082] 1) Data acquisition: At present, the message functions of various online government inquiry platforms are similar. The message content page on the online government inquiry platform usually contains the title, content, message type and reply after relevant processing. The data acquisition stage mainly obtains the content of the message and the corresponding government reply data from the online government inquiry platform. At the same time, it is necessary to obtain the type of message in order to filter out the complaint-type government inquiry messages studied in this invention.

[0083] 2) Data preprocessing: The message data obtained in the previous step is preprocessed in two aspects: text filtering and data cleaning: First, the text is filtered. Since the present invention extracts the elements of complaint messages, it is necessary to first filter out the text data of the complaint type. The present invention first filters out the complaint-type political messages according to the message tag information, but there are still some messages that are essentially consultation-type messages. Therefore, the present invention removes non-complaint-type messages from the remaining messages by manual inspection, and removes messages that have been replied but the reply content is being processed. Due to the differences in the complexity of the complaints of the masses, the masses and the processing department also have differences in the degree of carefulness and meticulousness when writing messages and replies. It can be observed that there is a huge difference in the length of messages and replies. Some messages or replies are less than 100 words and some messages or replies are more than 1,000 words. Considering that the large language model has a significant performance drop when the length is extrapolated and the model needs to use more computing power and video memory space when processing long texts, the text only retains messages with a text length of no more than 500 words in the message part and a text length of no more than 600 words in the reply part. Then the data is cleaned. According to the requirements of the model output format in the instructions used in the present invention, the model will use half-width symbols to segment the output content when outputting to distinguish different elements, element contents and element types. Therefore, the present invention unifies the punctuation used in messages and replies, replaces half-width punctuation with full-width punctuation, and uses spaces with ASCII encoding of 32 to replace invisible characters except line breaks. At the same time, unnecessary spaces and line breaks are deleted. Unifying punctuation marks can not only avoid the confusion caused by using the same symbols in the text and the output format, but also unifying symbols with the same meaning can avoid difficulties when the model processes different symbols with the same meaning.

[0084] 3) Data annotation: According to the requirements of entity element categories described in Table 2.1 and the corresponding relationship between the processing departments and processing measures in the text, the pre-processed message and reply data are manually annotated. Since the message part text is an informal text written by the masses, there are cases where the language is not standardized and colloquial. Therefore, in the annotation process, when the original language is not standardized, uses rhetorical questions, and lacks structure, it is necessary to appropriately modify the original text, add or delete redundant content as the content of the element, so that the annotated element content is complete and the expression is clear and concise. For example, in the message "…the outdoor billboard of the Maji Paradise shopping mall next to the north gate of Century Landscape Community (near the city government) turns on its lights every night until the next day. Because the light from the opposite side is too bright and shines directly into the room and living room, even with two layers of curtains drawn, it cannot be blocked, which seriously affects the children's sleep and work schedule...", the complaint of the message user is that "the outdoor billboard is too bright", but the message user also detailed the location of the billboard, lighting time and other information. Therefore, when extracting the complaint problem elements of the message, this part of the content needs to be appropriately deleted to ensure the simplicity and clarity of the extracted elements. Since the labeling task of the present invention is not a traditional entity recognition and relationship recognition task, in order to improve labeling efficiency, the present invention uses the open source data labeling platform Label Studio to perform data labeling. The platform allows users to customize the labeling interface and labeling targets to meet personalized labeling needs. The labeling results are exported in json format.

[0085] 4) Inject instruction text: Process the exported annotation data according to the instruction data template in Table 2.2. For each pair of messages and replies, add entity recognition instructions and relationship recognition instructions, and replace the "entity type list" with a list of eight message entity elements extracted from the text. Replace the "relationship type list" with "execute".

[0086] 5) Adjust the order of elements: Finally, in order to guide the model to generate in the order in which the elements appear in the original text, sort the elements or relationship triples in the answer part of the dataset. The elements in the answer are sorted according to the order in which the element content appears in the original text. The relationship triples in the answer are sorted based on the order in which the treatment measures appear in the original text as the main sorting basis, and the order in which the treatment departments appear in the original text as the secondary sorting basis. In order to represent the entity elements concisely and clearly, some annotated elements do not have exactly the same text in the original text. Therefore, the similarity between the element and any substring of the original text based on the Levenshtein distance is calculated, and the position of the substring with the highest similarity is taken as the position of the element. The sorted dataset is used as the dataset for extracting instructions for elements in government messages.

[0087] (2) Construction of Chinese information extraction instruction dataset: Studies have shown that fine-tuning a large language model using instruction data from multiple different tasks can improve the model's few-sample capabilities in other tasks. The present invention chooses to modify the existing public information extraction dataset to construct a Chinese information extraction instruction dataset, which is used to pre-train the model with information extraction instructions to improve the model's general information extraction capabilities and ability to understand extraction instructions. Next, the process of constructing the Chinese information extraction instruction dataset will be introduced, including data acquisition, preprocessing, annotation, and output format.

[0088] 1) Data acquisition: This paper collected 9 different data sets of different fields and different extraction tasks. The basic information of these data sets is shown in Table 2.3. Among them, the CMeIE data set is used to verify the effect of information extraction instruction pre-training under different training set scales, so it is not used to construct the Chinese information extraction instruction data set. However, it is processed according to the instruction template like other data sets.

[0089] Table 2.3 Basic information of the collected Chinese information extraction data set

[0090]

[0091] 2) Data preprocessing: The present invention first cleans the data collected in the Chinese information extraction dataset, mainly deleting data containing event categories, relationship categories, and entity categories that are not clearly defined, for example, "other" types of events in the CCKS2021 dataset, and "other" types of entities in the CMeIE dataset. Because different datasets use different data formats to save data, the present invention converts the formats of other datasets with reference to the format of the CCKS2022 dataset to facilitate subsequent unified processing. Taking into account the large differences in the number of samples in different datasets, the present invention limits the samples of each dataset to less than 10,000 with reference to the processing of Wang et al. If the number of samples is more than 10,000, 10,000 samples will be randomly sampled, where the CCKS2022 dataset contains multiple different extraction tasks, and each extraction task sample of the dataset retains 10,000 samples. The CMeIE dataset is not used to construct the Chinese information extraction instruction dataset, so the training samples are not deleted.

[0092] 3) Inject instruction text: Process the preprocessed data set according to the two-stage information extraction instruction template described in Section 2.1.1, add the instructions corresponding to the data set task, and replace the "type list" in the instruction template with the list of specific types in the data set task. Due to the limitation of the length of the model input text, the present invention splits the data samples with a total length of more than 2000 words for the three parts of instruction, text and answer.

[0093] After the above processing, the present invention constructs a Chinese information extraction instruction dataset for information extraction pre-training from the public Chinese information extraction dataset. The dataset has a total of 91,552 samples, including 50,903 entity recognition task samples, 23,214 relationship extraction task samples, and 17,435 event extraction task samples. An example of the format of the processed relationship extraction task sample is as follows:

[0094]

[0095] 2.3 LoRA-based method for fine-tuning instructions of the base model for feature extraction: In order to improve the extraction capability of the base model for feature extraction when the number of training samples is small, this aspect will use the Chinese information extraction instruction dataset and the political message feature extraction instruction dataset to fine-tune the model in turn. Both fine-tunings use the form of supervised fine-tuning commonly used in instruction fine-tuning research to train the model. Taking into account that full parameter fine-tuning requires a lot of storage and computing power resources, the present invention uses the LoRA method to fine-tune the model. When using the LoRA method to perform feature extraction training on a large language model, it is first necessary to convert the instruction data into a token sequence that the model can handle. Secondly, it is necessary to add an adapter layer to the model and freeze other parameters. Finally, the model is fine-tuned in a supervised manner.

[0096] (1) Training data preparation: In a large language model, a token represents the smallest unit that the model can process, usually a word, character, punctuation mark or phrase, and each token has its own unique digital number. Each large language model has a corresponding vocabulary that records all tokens that the model can process and their numbers. The input that the model can process is a sequence composed of token numbers, hereinafter referred to as a token sequence. Therefore, before training, the training text needs to be converted into a token sequence that the corresponding model can process. At the same time, because the present invention uses a two-stage information extraction instruction design, it is necessary to use the multi-round dialogue capability of the large language model. Therefore, it is necessary to add tokens that identify the dialogue role and the start and end positions before and after the token sequence of questions and answers so that the model can correctly understand the historical dialogue when it is generated. Taking relation extraction as an example, the training data S of a relation extraction task can be represented as the five parts shown in formula (2.1) in the instruction data set. First, the text of each part is segmented according to the vocabulary of the model and replaced with the corresponding token number. Then, dialogue-related tokens are added. The present invention combines the instructions in the instruction data set with the corresponding text as the questions of the dialogue, and uses the answers in the instruction data set as the answers to the dialogue, so as to use the same format as that used for model training when fine-tuning the instructions. At the same time, "Text to be processed:" is inserted between the instructions and the text in the instruction data set so that the model can distinguish between instructions and text. There are differences in the dialogue-related token formats of different models. Taking the Qwen model as an example, the structure of the processed training data S' is shown in Table 2.4.

[0097] Table 2.4 Example of token sequence format of relation extraction training data in the Qwen model

[0098]

[0099]

[0100] (2) Model preparation: Existing studies have shown that when training a model on a specific task, it only needs to be trained in a solution space with a number of dimensions far lower than the number of model parameters to converge. Therefore, many tasks have a low intrinsic dimension. The LoRA method limits the dimension of the solution space by adding a low-dimensional adapter layer to the weight matrix of the linear layer and freezing the original weights, thereby reducing the number of trainable parameters and reducing the storage and computing power requirements during training without affecting the fine-tuning effect. The existing large language model can generally be divided into three parts in terms of structure: an embedding layer, multiple Transformer decoding layers, and a language model output layer. The embedding layer is used to convert the input token sequence into the corresponding embedding vector, and multiple Transformer layers are used to capture the semantic features of the token in the input sentence and generate an embedding vector containing contextual semantic information for the input token. The language model output layer calculates the probability that the embedding vector containing contextual semantic information belongs to each token in the vocabulary.

[0101] Since the feature extraction task does not require adding new tokens to the model, all parameters of the embedding layer and the language model output layer are frozen, and the adapter layer is not added. While reducing the number of trainable parameters, it is possible to avoid semantic shift caused by adjusting the embedding parameters and affecting the training effect. For the Transformer decoding layer, the research of Hu et al. shows that under the same trainable parameters, reducing the adapter dimension and adding adapters to more linear layers can improve the training effect of the model. The experimental results show that the weight W of the multi-head attention part in the Transformer decoder layer q ,W k ,W v ,W o Adding adapters can achieve the best results. Therefore, in addition to the multi-head attention part, this aspect also adds adapters to the two linear layers in the feed-forward network part of the Transformer decoder layer.

[0102] Research shows that by merging the LoRA adapter layer and adding a new adapter layer, the model can search in different solution spaces. Since the model needs to be fine-tuned using two different datasets, and considering the differences in domains, text sources, etc., different information extraction tasks also have some different capability requirements for the model, so this aspect adds an adapter layer before each fine-tuning and merges the adapter layer after each fine-tuning, so that the model can search for the optimal weights in different solution spaces when fine-tuning different tasks, so as to improve the model's performance in extracting political message elements.

[0103] (3) Supervised fine-tuning: After converting the instruction dataset into the corresponding token sequence and adding the adapter layer to the model, the model can be trained using the supervised fine-tuning method. Specifically, suppose a token sequence of training data is T = {t1, t2, …, t n}, where n is the number of tokens in the token sequence, t i Indicates the number corresponding to the i-th token.

[0104] First, the token sequence T is input into the model after adding the adapter layer. After the embedding layer, multiple Transformer decoding layers, and the language model output layer, the model will output Where vocab_size is the total number of tokens in the model vocabulary. L represents the cosine value of the embedding vector of each token in T in the input sentence and the standard output embedding of all tokens in the output layer of the language model. The matrix L is processed row by row by softmax to obtain the matrix L'. L' represents the conditional probability distribution of the token generated by the next position of each token in T, that is, the i-th row and j-th column of L' represents the conditional probability of generating a token numbered j from the i+1-th token after the first i tokens of T. Assume that each t i Each corresponds to a probability distribution t' i , t' i The random variable of is the value of all tokens in the model vocabulary, of which only the token numbered t i The probability of the token is 1, and the probability of other tokens is 0.

[0105] Then, calculate the loss function. The cross entropy loss function is usually used when fine-tuning a large language model. The text is partially adjusted based on this. The specific calculation formula is:

[0106]

[0107] where t i+1 Represents the i+1th token in the input sequence T, L' i Represents the conditional probability distribution of the i+1th token. CrossEntropyLoss represents the cross entropy loss function. Because the input contains the instructions, texts, and answers of the instruction dataset, and in the feature extraction task, only the ability of the model to generate correct answers needs to be trained, if the loss function of all positions is calculated, not only will the model be trained to generate corresponding answers based on the instructions and texts, but also the model will be trained to generate instructions or texts. In order to reduce the interference of these trainings on the information extraction ability of the model, it is stipulated that q is used only when and only when the text corresponding to the i+1th token belongs to the answer part or the dialogue termination mark of the answer part. i+1 =1, otherwise qi+1 = 0. Finally, the gradient of the parameters in the model is calculated according to the loss function and the weights of the trainable parameters are updated.

[0108] 2.4 Experiment and results analysis

[0109] 2.4.1 Experimental design

[0110] (1) Experimental data: This experiment uses the following three datasets.

[0111] 1) Instruction data set for extracting elements of political messages: The present invention selects to collect the data required for research from the "Asking about Hunan" online political inquiry platform. As a provincial political inquiry platform, the platform was founded earlier and has a large number of users. A large amount of political message data has been accumulated in the platform. At the same time, the platform provides message types and subject information to facilitate subsequent screening of the required message data. According to the production steps of the political element extraction instruction set in Section 2.2.2, the present invention first uses Python language to write a web crawler code to crawl the HTML code of the message page from the "Asking about Hunan" platform, and then uses XPath to locate the HTML element position of the message title, message content, reply content, message category, theme and other contents in the message page, then reads the text content of the HTML element, and finally saves it to the SQLite database. The present invention crawls a total of 319,774 messages and reply data that can be displayed on the website with message IDs between 1 and 600,000 on the "Asking about Hunan" platform. The question time of these messages is between 2010 and 2022, and 85,585 messages contain information such as replies and message types. Among these messages, there are 35,591 messages whose message type is displayed as "complaint" on the platform, and the message time is between August 6, 2018 and August 9, 2022. After randomly sorting these messages and processing and annotating them according to the steps in Section 2.2.2, the present invention obtains a total of 982 annotated complaint-type political inquiry messages and corresponding replies, including 7,354 entity elements and 1,703 corresponding relationships between processing departments and processing measures. Example of the processed message and reply data format:

[0112]

[0113] The number of entity elements of different categories is shown in Table 2.5. Before the experiment, the dataset was divided into training set, validation set and test set in a ratio of 7:1:2. The dataset of extracting elements from government messages will be referred to as D-Elecment dataset.

[0114] 2) Chinese information extraction instruction dataset, which contains data from multiple information extraction tasks in multiple fields. For details, please see Section 2.2.2, hereinafter referred to as the D-pretrain dataset.

[0115] 3) CMeIE dataset, which is a Chinese medical text entity relationship extraction dataset, including pediatrics and 100 common disease corpora. The CMeIE dataset used in this experiment has been processed according to the instruction model, see Section 2.2.2 for details. It should be noted that since the validation set of the CMeIE dataset has no public labels, in this experiment, the validation set is randomly split into two parts, used as the validation set and the test set respectively.

[0116] Table 2.5 Number of entity elements of different categories

[0117]

[0118]

[0119] (2) Experimental model: This experiment uses three different large language models to verify the effectiveness of the method for constructing a factor extraction base model based on large language model instruction fine-tuning proposed in the present invention. The models used in this experiment are: 1) ChatGLM3 model. ChatGLM3 is a dialogue pre-training model jointly released by Zhipu AI and Tsinghua University KEG Laboratory. This model is trained based on the general language model (GLM) architecture and has the characteristics of smooth dialogue and low deployment threshold. 2) Qwen model. Qwen is an open source pre-trained language model released by Alibaba. The model has stably trained large-scale, high-quality and diverse data, covering multiple languages, and has the capabilities of chatting, text creation, summarization, information extraction, translation, etc. 3) Yi model. The Yi model is a bilingual language model trained on the 3T language corpus and performs well in language cognition, common sense reasoning, reading comprehension, etc. The basic parameters of the model used in this experiment are shown in Table 2.6.

[0120] Table 2.6 Basic parameters of the model

[0121]

[0122] (3) Experimental parameters: The linear layers of the three large language models in this experiment with the adapter layer added are shown in Table 2.7.

[0123] Table 2.7 Linear layer with added adapter layer

[0124]

[0125] Through preliminary exploratory experiments, the parameter settings for pre-training the model for information extraction tasks using the D-pretrain dataset are shown in Table 2.8. Because this dataset has a lot of data, the model can be fully trained when Epochs is 4. When training the model using the D-Elecment dataset, Epochs is 12 and the other parameter settings are the same as in Table 2.8.

[0126] Table 2.8 Fine-tuning parameter settings for information extraction instruction pre-training using the D-pretrain dataset

[0127]

[0128] The model's decoding strategy uses a search strategy to generate deterministic results, and the optimal sequence retained at each step is set to 2 to avoid consuming too much computing power and storage space during the decoding process.

[0129] 2.4.2 Evaluation indicators: In information extraction tasks, precision, recall and Micro-F1 indicators are generally used to evaluate the effect of classification models. Assume that the label of the dataset is Among them l i Represents the basic unit set of the i-th sample label, It represents the basic unit of information that needs to be extracted from the jth sample of the i-th sample. The extraction result of the model is where l' i represents the basic unit set extracted by the i-th sample model, Represents the jth basic unit extracted by the i-th sample model. The granularity of basic units varies for different information extraction tasks. In the named entity recognition task, the basic unit represents a binary consisting of an entity and its relationship. In the relationship extraction task, the basic unit represents a triple consisting of a relationship type, subject and object. In the event extraction task, the basic unit represents the type of an event and a set of all its arguments and their corresponding categories. The same basic unit is then used to represent the extraction result of the model. Depending on whether the basic unit of the label is consistent with the basic unit extracted by the model, the extraction result and the label of the model can be divided into two parts. The total number of basic units in the extraction result that are exactly the same as those in the label is recorded as TP, the total number of basic units that appear only in the extraction result is recorded as FP, and the total number of basic units that appear only in the label is recorded as FN.

[0130] The precision rate indicates the proportion of correct basic units extracted from the model extraction results. The calculation formula is as follows:

[0131]

[0132] The recall rate indicates the proportion of basic units correctly extracted from the label, and the calculation formula is as follows:

[0133]

[0134] Micro-F1 is a comprehensive indicator that combines precision and recall to reflect the overall performance of the model. Its calculation formula is as follows:

[0135]

[0136] Since the elements in the question and answer message element extraction instruction dataset are modified for the elements that are unclear, missing or redundant in the original text, the model cannot simply copy part of the text in the original text for generating elements, but needs to modify the text appropriately to make it more complete and clear. However, for the elements that are relatively complete and clear in the original text, the model often makes certain modifications during generation, and these modifications usually do not affect the semantics. Therefore, the performance standard of the model evaluated on the question and answer message element extraction instruction dataset using the Micro-F1 indicator is too strict. The basic unit extracted by the model must be completely consistent with the label before it is included in the TP, which cannot effectively distinguish the differences between models. Gui et al. also noticed this problem when designing the evaluation indicators of the instruction information extraction dataset InstructIE, so they added the Rouge-2 indicator commonly used in sequence generation tasks. When Jiao et al. used the instruction fine-tuning method to train the model to complete the on-demand information extraction task (On-DemandInformation Extraction Task), they chose to use the SentenceBERT model to calculate the similarity of the header part to calculate the F1 value as the evaluation indicator, and used the Rouge-L F1 indicator to evaluate the accuracy of the model on the table content.

[0137] This experiment also uses a similar soft matching method to calculate precision, recall, and Micro-F1. Since the SentenceBERT model does not have the ability to process Chinese text, this experiment uses the text2vec-large-chinese model in the text2vec toolkit to calculate sentence similarity. Specifically, for a basic unit extracted by the model Divide it into parts representing types Such as entity categories, relationship categories, etc., as well as parts that represent specific content Such as the value of the entity, the value of the subject, etc. Similarly, the basic unit in the tag Divided into type parts and the content section when A basic unit in the label of the same sample When the type part is exactly the same and the sentence similarity of the content part is greater than or equal to the threshold p, it is considered and is a match and should be counted in TN. Note that for the relation extraction task, the content part of the basic unit includes the subject and the object. It is considered a soft match if and only if the sentence similarity of the subject and the object is greater than the threshold. Any basic unit in the label of the same sample When none of the is the basic unit of extraction error and is counted in FP. Any basic unit in the extraction results of the same sample When neither of them constitutes a soft match, it is considered It is a basic unit that is not correctly extracted and is counted in FN. Obviously, when the similarity threshold p = 1, only basic units with the same type and content are matched, which is called strict matching. Then, the formula is used to calculate the precision, recall and Micro-F1. In the following, Pp, Rp and F1-p will be used to represent the precision, recall and Micro-F1 when the similarity threshold is p.

[0138] 2.4.3 Experimental results and analysis: In order to explore the effectiveness of the proposed method for constructing a base model for feature extraction based on fine-tuning of large language model instructions, the text classification model is fine-tuned using the training set of the D-Elecment dataset with and without the D-pretrain dataset for information extraction tasks, and then tested on the test set of the D-Elecment dataset. The results of entity feature extraction are shown in Tables 2.9 and Figure 3 As shown in Table 2.10 and Figure 4 shown.

[0139] After using the D-pretrain dataset for pre-training of information extraction tasks, the entity factor extraction effects of the three models are improved. When the similarity threshold is 0.8, the precision rate is increased by an average of 1.75%, the recall rate is increased by an average of 1.4%, and the F1 value is increased by an average of 1.6%. In the relationship factor extraction task, the effects of the three models are improved after using the D-pretrain dataset for pre-training of information extraction tasks. When the similarity threshold is 0.8, the precision rate is increased by an average of 1.26%, the recall rate is increased by an average of 1.09%, and the F1 value is increased by an average of 1.24%. The effectiveness of the method proposed in the present invention is verified. When the scale of training data is small, using labeled data in other fields of information extraction tasks for pre-training can help improve the general ability of the model in performing information extraction tasks, thereby improving the model's ability to extract political message elements.

[0140] Compared with the entity element extraction task, the effect of the relational element extraction task is poor. Through the analysis of the extraction result data, there are two main reasons for this gap. First, in the reply to the message, there are complex situations such as multiple processing departments and processing measures appearing alternately, and multiple processing departments taking certain processing measures after being delegated step by step. It is difficult to identify the corresponding relationship between processing departments and processing measures. Second, the relationship recognition subtask of relational element extraction uses the relationship recognition instruction to control the model to generate the corresponding triples based on the element extraction results. Although it alleviates the problem of the model generating the wrong processing department or processing measures when directly generating triples, the relationship recognition result is limited by the recall rate of the extraction of processing departments and processing measures, and there is a certain error propagation problem.

[0141] Table 2.9 Entity feature extraction task test results (unit: %)

[0142]

[0143] Table 2.10 Relationship factor extraction task test results (unit: %)

[0144]

[0145] Since the scale of the D-Elecment dataset is relatively small, this experiment uses the CMeIE dataset with a larger number of samples to test the impact of information extraction task pre-training on the model's information extraction task performance in a specific field under different training set scales. First, the Qwen model that has been pre-trained for information extraction tasks and the Qwen model that has not been pre-trained for information extraction tasks are fine-tuned using the complete CMeIE dataset training set, and then 10% of the data is randomly extracted from the training set of the CMeIE dataset as a training set for fine-tuning. At the same time, according to the instructions of the CMeIE dataset, the Chinese-Roberta-WWM-Ext-Large model is used to fine-tune the training data of two scales as a reference baseline. Because all the entities in this dataset are words or phrases in the original text, a strict matching method with a similarity threshold of 1 is used to calculate the precision, recall, and F1 value as evaluation indicators. The experimental results are shown in Table 2.11.

[0146] Compared with the chinese-roberta-wwm-ext-large model, the Qwen model without information extraction task pre-training performs slightly better when using the complete data set, mainly due to the improvement in precision. The accuracy and recall rates are significantly improved when using a small-scale training set. The Qwen model with information extraction task pre-training has improved on both scales of training sets compared to the model without information extraction task pre-training, and the improvement is more obvious when using a small-scale training set. Among the three models, the Qwen model with information extraction task pre-training has the least performance degradation when using a small-scale training set, indicating that information extraction task pre-training helps the model learn more general information extraction capabilities and improves the model's information extraction capabilities in the target field, especially when the target field dataset is small.

[0147] Table 2.11 Relationship extraction task test results of CMeIE dataset (unit: %)

[0148]

[0149] 3. Factor extraction method based on evidence theory and integrated learning: A method for constructing a factor extraction base model based on fine-tuning of large language model instructions is proposed above, and experiments have verified that this method can train a model with good extraction effect on a small-scale data set. However, in the process of using a large language model to generate factors, errors in the generation of preceding factors will affect the generation of subsequent factors, causing error transmission. Using a single model for factor extraction lacks a method to identify errors in preceding factors, making it difficult to solve this problem. In response to this problem, the present invention proposes a factor extraction method based on evidence theory and integrated learning. This method uses multiple models trained in the previous chapter to extract multiple candidate factors and uses these models to evaluate all candidate factors, and then uses the evidence reasoning rule method to fuse the evaluation results to determine the extracted factors. This method can effectively identify and select factors with high probabilities on multiple models, avoid erroneous results due to the bias of a single model, and thus alleviate the problem of error transmission.

[0150] 3.1 Framework of the factor extraction method based on evidence theory and integrated learning: The present invention uses a sequence generation form to extract factors from government messages. If the model generates an erroneous pre-order factor, in the process of generating the subsequent factors, the model will refer to the input instructions, text and pre-order factors to generate the subsequent factors. In particular, there is a dependency relationship between the entity factors such as the complaint problem, the location of the complaint problem, and the appeal extracted by the present invention. Therefore, it is necessary to check the factors generated by the model and correct the erroneous factors. The present invention uses multiple different large language models to evaluate the generated factors to identify the erroneous factors, and uses a search-based candidate factor search method to generate multiple candidate factors to provide a larger selection range to provide better candidate factors. Unlike traditional large language model integration research that integrates after the model has completely generated results, the present invention chooses to integrate immediately after the model generates an factor, and continues to the next factor after the integration result, so that the model that originally generated the erroneous factor can continue to generate on the correct result after integration, reducing the problem of error transmission. The framework of the factor extraction method based on evidence theory and integrated learning is as follows: Figure 5 This integration method is a cyclic process. In each cycle, multiple candidate elements are searched and one of them is selected as the result and added to the input text. The calculation process of each round in the cycle is as follows:

[0151] First, a search strategy is used to use different models to search for several candidate elements after the instructions, texts and extracted elements of the same sample. The set of all candidate elements is used as the selection range for this round of integration. The candidate element search method is detailed in Section 3.2. Then calculate the confidence of each model for each candidate element. The confidence of each model for all candidate elements participates in the evidence fusion of this round as a piece of evidence. The candidate element confidence calculation method is detailed in Section 3.3. Then calculate the weight of each model. The present invention uses the NSGA-II algorithm to determine the weight of each model and use it as the weight of the evidence generated by the model. For relevant details, please refer to Section 3.4. Finally, the evidence reasoning rule method is used to fuse the confidence of different candidate elements of different models, select the candidate element with the highest probability after fusion as the element generated in this round, add it to the extracted elements, and then generate the next round of elements. The evidence fusion calculation method based on evidence reasoning rules is detailed in Section 3.5.

[0152] In order to terminate feature generation at the appropriate time, the confidence of the termination token generated by each model is calculated in each round of model evaluation, and it is used as a special factor to participate in the fusion of the results. When the factor with the highest confidence in the fusion result is the termination token, the feature generation is stopped. The pseudo code of the feature extraction method based on evidence theory and ensemble learning is shown in Algorithm 3.1.

[0153]

[0154] 3.2 Search-based candidate element search method: In evidence theory, the recognition framework represents a set of mutually exclusive propositions that can constitute a complete set. For entity element extraction tasks and relationship element extraction tasks, a proposition represents the next element generated after instructions, texts, and generated elements. The present invention uses a sequence generation method to implement element extraction. The number of all possible elements is an exponential based on the size of the model vocabulary. It is not feasible to obtain all possible generated elements in terms of computational cost. However, most possible elements are just random combinations of words without practical meaning, and are usually not considered as possible answers to questions. From the perspective of a large language model, the generation probability of most possible generated elements is extremely low. Therefore, the present invention only selects a set of elements with high conditional probabilities as the recognition framework. Specifically, the present invention uses a search strategy to generate multiple elements through multiple models respectively, and uses these elements and a set of tokens representing termination as the recognition framework. As described in Section 2.2, the generation formats of entity elements and relationship elements are basically the same. Unless otherwise specified, the candidate elements in the following text refer to both entity elements and relationship elements. The present invention uses a search decoding strategy to generate multiple candidate elements. This strategy can search for multiple candidate elements with high conditional probabilities at a relatively low computational cost. Although the use of sampling-based Top-K, Top-p and other decoding strategies can also generate multiple different candidate elements by performing multiple samplings, it is also possible to generate multiple identical candidate elements, resulting in a waste of computing resources. At the same time, the characteristics of the task of extracting elements from government messages are taken into account.

[0155] (1) Since in the search algorithm, the criterion for evaluating the generated text sequence is the conditional probability of the text sequence, when using traditional Chinese characters or synonyms to replace words in the text, the conditional probabilities of the two text sequences may not differ much. Therefore, it is easy to generate multiple slightly different versions of the same selected element using the classical search algorithm. To better generate multiple different candidate elements, the present invention prohibits the model from generating any traditional Chinese characters. Although the question-and-answer message data used does not contain any traditional Chinese characters, when using classical search decoding, the model may generate candidate elements containing traditional Chinese characters. Since the vocabulary of the model used in the present invention covers common Chinese characters, Chinese words, and English words, among which the Chinese characters and Chinese words include simplified Chinese characters and traditional Chinese characters. In the pre-training of massive data, the model can learn the relationship between traditional Chinese characters and their corresponding simplified Chinese characters. Specifically, in the output embedding matrix of the output layer of the model, the embedding vectors of simplified Chinese characters and their corresponding traditional Chinese characters have a high cosine similarity. Therefore, during the search process, elements represented by simplified Chinese characters and elements represented by traditional Chinese characters usually have similar conditional probabilities. As a result, the search results may include multiple candidate elements that have no difference except for traditional and simplified Chinese characters. For example, when using the Qwen model for search, it will search for two candidate elements: "Daokou Village, Changpu Township, Suining County, the location where the complaint problem occurred" and "Daokou Village, Changpu Township, Suining County, the location where the complaint problem occurred". In the output embedding of the output layer of the Qwen model language model, the cosine similarity between the output embeddings corresponding to the token "鋪" and "铺" is 0.6172. Therefore, the present invention skips all sequences containing traditional Chinese characters at the stage of selecting and retaining sequences during the search process to avoid this problem.

[0156] (2) Only generate the element content part of the entity candidate element. This modification is only used when extracting entity elements because there is only one type of relationship in the relationship element extraction task and it is located at the forefront of the relationship triple. As described in Section 2.2.2, the output format of an entity element is "(element content, element category)". During the model training process, the 8 entity element categories frequently appear as part of each entity element. Therefore, when the model outputs the token sequence corresponding to the correct entity element category, the conditional probability of each token is very high. In the search algorithm, the length of each candidate sequence is exactly the same. Therefore, if the element content of some sequences is shorter but the element category is longer, its conditional probability may be higher than that of other sequences with longer element content. To reduce the influence of the element category length on the sequence score, the present invention adopts the following strategy during the search: After a sequence completes the generation of the element content part, that is, when the last token of the sequence is the token corresponding to a comma, stop the generation of the sequence. Then, calculate the confidence of generating each category after this sequence and select the category with the highest confidence as the category of this candidate element. The calculation method of the element category confidence is described in detail in Section 3.3.

[0157] The process of searching for candidate entity elements using the modified search algorithm is shown in Algorithm 3.2. The algorithm needs to input the sample's instructions, text, and generated elements, and specifies the number of candidate elements to be searched, num_beams, by setting the model hyperparameters. At the same time, the initial value of the number of unfinished candidate sequences k is set to num_beams; the initial value of k unfinished sequences is set to an empty sequence. First, the model is used to predict the probability distribution of the next token of the current unfinished candidate sequence in the model vocabulary, and the sequence that does not contain traditional Chinese characters is screened out from the token sequence composed of the unfinished candidate sequence and the next token. Next, the conditional probability of the last token of the screened token sequence is multiplied by the conditional probability of the corresponding original sequence (unfinished candidate sequence) as the conditional probability of the new token sequence (the conditional probability of a token sequence of length k = the conditional probability of the sequence composed of the first k-1 tokens of the sequence × the conditional probability of the last token.). Then, the k sequences with the highest conditional probabilities are selected from the new token sequence as the unfinished candidate sequences for the next step. Finally, check whether there is a token sequence whose last token is a comma in the candidate sequence of the next step. If so, calculate the confidence of each feature category generated after these token sequences (the token sequence of the feature name), select the feature category with the highest confidence as the feature category, mark these token sequences as completed, and then reduce the value of k accordingly. If the number of unfinished sequences k is still greater than 0 at this time, continue to loop until all sequences are completed.

[0158] The process of searching for candidate relationship elements is similar to that of searching for candidate entities. The difference is that when searching for candidate relationship elements, the condition for sequence generation is that the last token of the sequence is a token containing a right bracket. Finally, check whether there is a token in the candidate sequence in the next step whose last token is any token in the set of tokens starting with a right bracket in the vocabulary.

[0159] It should be noted that when using a large language model to generate text, it usually stops after the model outputs a termination token. However, the present invention does not generate a termination token in the candidate element search method, but adds a termination token in each round of integration and calculates its confidence to determine the timing of terminating the element generation. Because the present invention uses three different large language models for integration, but the vocabulary of these three models is not the same, the same text sequence will be divided into different token sequences in different models. Take the three punctuation marks “),(” between two elements as an example. In the Qwen model, there is one token “),(”, in the ChatGLM3 model, there are two tokens “),” and “(”, and in the Yi model, there are three tokens “)”, “,” and “(”. This means that when generating candidate elements after some elements have been generated, it is necessary to start generating after the type of the last element of the generated elements, so that the Qwen model and the ChatGLM3 model can generate the termination token in the normal order. However, starting to generate after the type of the last element of the generated elements means that other content may be generated in the feature type part of the last element of the generated elements, and additional tokens need to be generated to increase the calculation time when generating candidate elements. Adding a termination token to the recognition framework can avoid these problems and reduce the complexity of the algorithm.

[0160]

[0161] 3.3 Candidate factor confidence calculation method based on conditional probability geometric mean: In evidence theory, basic probability distribution represents the degree of support of an evidence for each proposition or proposition combination in the power set of the identification framework. For the task of extracting elements from political messages, a piece of evidence is the degree of support for each candidate element by a large language model involved in the integration. In the search decoding strategy, the conditional probability of generating a text sequence is used as the basis for evaluating the quality of the generated sequence. During the search process, the sequences to be evaluated have the same length. However, when evaluating the quality of candidate elements with different lengths, using conditional probability as the evaluation basis will obviously benefit candidate elements with shorter texts. The large language model is in the sequence t1, t2, …, t q The sequence t generated later q+1 ,t q+2 ,…,t q+u The conditional probability is:

[0162]

[0163] The conditional probability of each token in the generated sequence is a number in the interval (0,1), so even if the conditional probability of each token is the same, the conditional probability of the long text will be lower than that of the short text. Therefore, the present invention selects the geometric mean of the conditional probability of each token in the candidate element as the confidence of the candidate element. The use of geometric mean can eliminate the influence of text length and can compare candidate elements of different lengths more fairly. Similarly, categories of different lengths will also affect the confidence of the candidate elements, because element categories frequently appear in the fine-tuning process, and the conditional probability of each token in the correct element category is close to 1. When the length of the element content part of the candidate element and the conditional probability are the same, the confidence of the candidate element with a longer text of the element type is higher. Therefore, the element type part of the candidate element will be regarded as a single token to participate in the confidence calculation, and its conditional probability is the geometric mean of the conditional probability of each token, reducing the influence of the type length on the confidence.

[0164] Specifically, for the entity feature extraction task, assume that the token sequence corresponding to the sample's instructions, text, and generated features is T, where T includes the comma and left bracket after the last generated feature to ensure that the same token as used in training is used to represent the three punctuation marks "),(" between the two features. Candidate entity feature θ e The token sequence of the element content part is Where lv is the sequence length. The token sequence of the feature type part of the candidate feature is Where lc is the sequence length. The set TE represents the set of tokens starting with a right bracket in the vocabulary of model m. Taking the Qwen model as an example, TE = {"), (",")"}. Then the candidate entity factor θ e The confidence calculation formula on model m is as follows:

[0165]

[0166] where t comma Indicates the token corresponding to the comma. PEC indicates the conditional probability of the token corresponding to the comma. PET indicates the geometric mean of the conditional probabilities of the candidate element type part. PEE indicates the probability of the token at the end of the element. As mentioned above, there may be two different tokens used to indicate the end of an element in the Qwen and ChatGLM3 models, so the sum of the conditional probabilities of the two is taken.

[0167] For the relational element extraction task, assume that the token sequence corresponding to the sample instructions, text and generated elements is T, and the candidate relational element θ r The token sequence consisting of the relationship type, subject, and object is Where lr is the sequence length. Then the candidate relationship element θ r The confidence calculation formula on model m is as follows:

[0168]

[0169] Where TRE represents the set of tokens starting with a right bracket in the vocabulary of model m.

[0170] In the basic probability allocation stage, in addition to calculating the confidence of the candidate elements, the confidence of the termination token also needs to be calculated. The present invention uses the conditional probability of the termination token to represent the confidence of the termination token, and the calculation formula is:

[0171] Confidence(m,θ end )=P(t end |T') (3.4)

[0172] where θ end is a proposition in the identification framework that indicates the termination of continued generation, t end represents the termination token, and T' represents the token sequence corresponding to the sample's instructions, text, and generated elements.

[0173] Finally, the confidence of all propositions in a piece of evidence on the corresponding model is scaled to ensure that the sum of the basic probability distribution is 1. Specifically, for model m and identification framework Θ = {θ1,θ2,…,θ n}, the evidence e corresponding to model m m for:

[0174]

[0175] where p m,θ Indicates evidence m The confidence in the proposition θ.

[0176] 3.4 Model weight determination method based on NSGA-II algorithm

[0177] When using multiple models for integration, it is necessary to assign appropriate weights to each model to adjust the influence of different models on the integration results, thereby improving the integration effect. Since the present invention includes two tasks, namely, factor extraction and identification of the correspondence between processing departments and processing measures, an algorithm with multi-objective optimization capabilities is required to search for model weights. Secondly, because the relationship between model weights and the effects of the final tasks is complex and there is an obvious mutual influence between the weights of several models, it is inconsistent with the premise assumptions of the hyperparameter search algorithm based on Bayesian optimization. Considering the low convergence efficiency of grid search and random search algorithms and the high computing power consumption of using multiple large language model integrations for factor extraction. Therefore, the NSGA-II algorithm with multi-objective optimization capabilities is used to search for the optimal combination of model weights in the validation set.

[0178] The NSGA-II algorithm is a non-dominated genetic algorithm used for multi-objective optimization tasks. It adopts an elite retention strategy to ensure that individuals with better performance can continue to participate in inheritance. At the same time, it uses the crowding distance indicator to determine the degree of crowding of individuals in the solution space, and gives priority to individuals in the relatively "open" area of ​​the solution space, so that the searched individuals can be more evenly distributed on the Pareto optimal frontier, thereby improving the diversity of the population. The key steps of the NSGA-II algorithm will be introduced below.

[0179] (1) Chromosome encoding method: In genetic algorithms, feasible solutions are regarded as chromosomes. Since the goal of using NGSA-II in the present invention is to search for the optimal weight combination, a floating point encoding scheme is used. Specifically, each gene position represents the weight of a model, represented by a floating point number. Considering that the sum of the weights of evidence should be 1, and the weight of the evidence is the weight of the model that generates the evidence, each gene position in the chromosome is divided by the sum of all gene positions at the end of operators involving the generation of new chromosomes, such as population initialization and the generation of new individuals, to ensure that the sum is 1. Chromosome encoding example:

[0180]

[0181] (2) Designing fitness function: In genetic algorithms, the fitness function is used to evaluate the performance of individuals in solving the target problem. Since the feature extraction task consists of two parts: entity feature extraction and relationship feature extraction, two fitness functions are needed to determine the effects of different weight combinations on the two tasks.

[0182] 1) For the entity feature extraction task, the present invention uses the F1-0.8 index of the integrated model on the validation set of the D-Elecment dataset as the fitness function of the entity feature extraction task. For detailed information about the F1-0.8 index, see Section 2.4.2.

[0183] 2) For the relational element extraction task, the present invention uses the F1-0.8 index of the integrated model on the validation set of the D-Elecment dataset as the fitness function of the relational element extraction task.

[0184] (3) Genetic operators: The genetic operators in the genetic algorithm mainly include selection operators, crossover operators and mutation operators, which are used to generate offspring populations from parent populations. The present invention chooses to use the binary tournament algorithm, randomly extracting two individuals from the parent individuals each time, and selecting the individual with the lower Pareto rank. When the Pareto ranks of the two individuals are the same, the individual with the larger crowding distance is selected, and the process is repeated until half of the number of individuals in the parent population is selected for reproduction. Please refer to the literature for the calculation method of the Pareto rank and crowding distance. Since the present invention chooses to use floating point encoding, the simulated binary crossover operator and the polynomial mutation operator are used to better search the surrounding area of ​​the weight with excellent effect.

[0185] (4) Iterative search using NSGA-II algorithm: After determining the genetic encoding method, fitness function and genetic operator, follow the steps of NSGA-II algorithm to perform population initialization, generation of offspring population, calculation of non-dominated level and crowding distance, sorting and selection of new parent population, etc. to search for a better model weight combination. For details of the search process, please refer to the literature. Considering that a large amount of computing power and time are consumed when generating a large language model, and considering the characteristics that decoding methods without random processes such as greedy search and search produce the same generation results when the same sequence is input, the present invention adds a cache mechanism to the NSGA-II algorithm search process, which is used to input text and corresponding record generation results to speed up the search process. Specifically, the present invention saves the relevant parameters and results of the two steps of candidate element search and candidate element confidence calculation into the database. Taking candidate element search as an example, the parameters that affect the search results include the specific model used for search, the text before the model starts searching, and the search strategy and its parameters. The present invention uses the set of these parameters as a key and saves the search results as a value into the database. Whenever you need to search for candidate features, first search the database to see if the same key exists. If it does, return the result directly. If it does not exist, call the model to calculate and save the key-value pair to the database before returning the result.

[0186] (5) Optimal solution selection: After iterative search, a set of Pareto optimal solutions and corresponding Pareto optimal frontiers can be obtained. The present invention selects the one with the largest sum of F1-0.8 indexes of the entity factor extraction task and the relationship factor extraction task as the final searched weight.

[0187] 3.5 Candidate factor information integration method based on evidence reasoning rules: In evidence reasoning rules, the identification framework is composed of a set of mutually exclusive propositions that can form a complete set. In the task of extracting elements from political messages, the identification framework is a set of all candidate elements and termination tokens. An evidence e is defined as:

[0188]

[0189] where p θ Represents the confidence of evidence e to a proposition or a combination of propositions θ. The present invention regards the confidence of a model to all candidate elements in the identification framework as one piece of evidence. The weight of evidence represents the judgment of the decision maker on the importance of different evidence relative to the evidence in the process of evidence fusion. The present invention uses the weight of the model that generates evidence as the weight of evidence. The reliability of evidence represents the ability of the information source of the evidence to provide a correct evaluation of the problem. In ensemble learning, the performance index of the base model can usually be used as the reliability of the evidence generated by the base model. Considering that in the present invention, the evidence is generated by a large language model, the reliability of the evidence should be determined by the ability of the large language model to generate correct results in the task. As can be seen from Section 2.4.3, the three large language models are similar in terms of precision, recall and other indicators. At the same time, when performing evidence fusion, the confidence of the evidence to the candidate element or relationship triple is calculated by the confidence. Therefore, the present invention calculates the average confidence of each element in the label of the validation set in the D-Elecment dataset in model m as the reliability of the evidence generated by model m.

[0190] Specifically, the evidence e generated by model m m Reliability m The calculation formula is as follows:

[0191]

[0192] Where Y represents the set of all elements in the labels of the validation set of the D-Elecment dataset, |Y| represents the number of elements in the set Y, and the confidence calculation method of the elements is detailed in Section 3.3.

[0193] Assuming that all base models search out n-1 different candidate elements in one round of integration, the identification framework Θ = {θ1,θ2,…,θ n}, (i≠j; i,j=1,2,...,n). Assume that there are lm different base models that calculate the confidence of the candidate elements in the identification framework and generate lm pieces of evidence e1,e2,...,e lm , the weight of the i-th piece of evidence is w i , reliability is r i . Then the evidencei The weighted confidence distribution g i for:

[0194]

[0195] where P(Θ) is a special element added to preserve the remaining confidence after weight discounting, and specifies any subset of the recognition frame θ∩P(Θ)=θ. Let e(i) represent the first i pieces of evidence e1, e2, ..., e i After the fusion, the evidence generated by all the lm basis models can be calculated using the following iterative formula:

[0196]

[0197] where g θ,e (i) represents the confidence of the weighted confidence distribution of the evidence e(i) after the fusion of the first i pieces of evidence on the subset θ of the identification framework. P(Θ),e(i) It represents the confidence of the weighted confidence distribution of the evidence e(i) after the fusion of the first i pieces of evidence on the element P(Θ). For the first piece of evidence, there is g θ,e(1) =g θ,1 .

[0198] Select the element with the highest probability in the evidence e(lm) generated by all lm basis models after fusion as the integration result. If the integration result is a candidate element, add it to the answer and continue to integrate the next element. If the integration result is a termination token, stop the loop and take the existing elements in the answer as the final extraction result.

[0199] 3.6 Experiment and result analysis

[0200] 3.6.1 Experimental design

[0201] (1) Experimental data: The dataset used in this experiment is the complaint-type political inquiry message element extraction instruction dataset, namely the D-Elecment dataset. For more information about this dataset, please refer to Sections 2.2.2 and 2.4.1. This dataset is constructed by manually annotating complaint-type political inquiry messages on the "Ask Hunan" online political inquiry platform. There are a total of 982 complaint-type political inquiry messages and corresponding replies, and they are randomly divided into training set, validation set, and test set in a ratio of 7:1:2.

[0202] (2) Evaluation indicators: In order to reflect the effectiveness of the factor extraction method based on evidence theory and ensemble learning proposed in this invention, this experiment uses the following evaluation indicators.

[0203] 1) The precision rate Pp of soft matching is used to reflect the proportion of correctly extracted basic units in the extraction result, and p is the soft matching threshold. 2) The recall rate Rp of soft matching is used to reflect the proportion of correctly extracted basic units in the label, and p is the soft matching threshold. 3) The Micro-F1 value Fp of soft matching is used. The Micro-F1 value is a comprehensive indicator that combines the precision rate and the recall rate to reflect the overall performance of the model, and p is the soft matching threshold. The calculation method of the above indicators is detailed in Section 2.4.2.

[0204] (3) Comparative methods: The present invention selects the following three methods for comparative analysis. At the same time, for the sake of convenience, EG-ERRule is used to refer to the factor extraction method based on evidence theory and ensemble learning proposed by the present invention.

[0205] 1) ChatGLM3, Qwen, Yi: A single large language model that uses the D-pretrain dataset for information extraction instructions and fine-tunes using the D-Elecment dataset, and is also used as the base model for the integration. 2) EG-Avg: A method that uses the averaging method for integration at the element granularity. 3) FG-ERRule: A method that generates all elements at once and uses evidence reasoning rules for integration.

[0206] (4) Parameter setting: Unless otherwise specified, all base models use the search-based candidate feature generation method described in Section 3.2, and the number of sequences retained in each step of the search process is 2. All methods that require model weights use the NSGA-II-based evidence weight determination method described in Section 3.4 to perform weight search on the validation set of the D-Elecment dataset. After preliminary exploratory experiments, the parameters shown in Table 3.1 are used for model weight search. At the same time, the search is stopped after the F1-0.8 index of the two tasks of entity feature extraction and relationship feature extraction has not improved for more than 10 generations. The reliability of the model is calculated according to the method described in Section 3.5, and the results are shown in Table 3.2.

[0207] Table 3.1 NSGA-II algorithm search parameters

[0208]

[0209] 3.6.2 Experimental results and analysis: To verify the effectiveness of the feature extraction method based on evidence theory and ensemble learning, the present invention uses different methods to perform entity feature extraction tasks and relationship feature extraction tasks on the test set of the D-Elecment dataset. The test results of the entity feature extraction task of all comparison methods and base models on the D-Elecment dataset are shown in Table 3.3. It can be seen from Table 3.3 that:

[0210] (1) All ensemble methods have significantly improved the precision, recall and Micro-F1 values ​​compared to the base models, which shows the effectiveness of using confidence to evaluate the quality of candidate features and to develop ensemble strategies based on this. Although the features generated by the base models may be wrong due to factors such as the small number of training set samples and the limited model size, the correct features usually have a high confidence on the base models. Candidate features with high confidence on all three base models are more likely to be correct features. The average confidence of the generated results of different methods or models on the base models is calculated on the validation set of the D-Elecment dataset. The results are shown in Table 3.4. The average confidence of the features generated by several ensemble methods using confidence as an indicator for evaluating the quality of candidate features is high, while the average confidence of the generated results of a single base model on other base models is slightly lower than the average confidence of the validation set labels on other base models, indicating that each base model has a certain ability to identify erroneous features. Therefore, integrating the confidence information of multiple base models can identify the generation errors of the base models and select appropriate candidate features as the ensemble results.

[0211] (2) Compared with the EG-Avg method, the EG-ERRule method proposed in this paper has a similar precision under a similarity threshold of 0.8, a recall rate increased by 2.57%, and a Micro-F1 increased by 1.34%. This shows that compared with the average method, the candidate factor integration method using evidence reasoning rules can more effectively integrate the uncertain evidence generated by different base models.

[0212] (3) Compared with the FG-ERRule method, the EG-ERRule method, which integrates at the element granularity, has a slightly lower precision, but a significantly improved recall rate, with the Micro-F1 value at a similarity threshold of 0.8 increasing by 1.6%. One reason for the significant improvement in recall is that the generation of elements is more flexible, and whether each element needs to appear in the final generated answer can be evaluated and integrated separately, rather than being selected as a whole with other elements. The EG-ERRule method generated a total of 1,460 different elements on the D-Elecment dataset test set. The EG-Avg method, which also integrates at the element granularity, generated 1,410 different elements, while the FG-ERRule method only generated 1,358 elements. Overall, the EG-ERRule method has a better extraction effect, proving that integration at the element granularity is more effective.

[0213] Table 3.3 Entity feature extraction task test results (unit: %)

[0214]

[0215] Table 3.4 Mean confidence of the generated results of different methods or models on the base model

[0216]

[0217]

[0218] The results of the relational factor extraction task of all comparison methods and base models on the D-Elecment dataset are shown in Table 3.5. From Table 3.5, we can see that: (1) In general, the ranking of each model on this task is basically consistent with that in the entity factor extraction task. On the one hand, the relation recognition subtask is performed after the entity recognition subtask, and the subject and object of the extracted triples come from the entity factor extraction task. Its results are affected by the quality of the results of the entity factor extraction task. On the other hand, the two tasks require the model to have some similar capabilities to complete, such as the ability to understand the role of part of the text in the entire message or message reply. Therefore, the model that performs well in the entity factor extraction task also performs well in the relational factor extraction task. (2) Compared with other methods, the FG-ERRule method integrated at the entire generated content granularity has higher precision and lower recall. On the one hand, this is due to the influence of the entity factor extraction results. On the other hand, when using confidence to evaluate the entire answer composed of multiple triples, if all correct triples have the same conditional probability and all incorrect triples have the same but lower conditional probability, then the confidence of the entire answer is positively correlated with the proportion of correct triplets in the answer. Therefore, using this integrated method is more likely to select answers that only have a small number of high-confidence triplets, resulting in a higher precision but lower recall in the final test results.

[0219] Table 3.5 Relationship element extraction task test results (unit: %)

[0220]

[0221] The entity element generation results of the EG-ERRule method proposed in this invention are statistically analyzed according to element categories. Figure 6 As shown. Figure 6It can be seen that: (1) The entity element extraction effect of the processing department category is the best. There are two main reasons for this result. First, the number of processing department elements is second only to the processing measures, and the model has more samples for training this category; second, the processing departments in the message replies are mainly government departments such as public security, street offices, urban management, and public enterprises and institutions such as water companies, state grids, and natural gas companies responsible for water supply, power supply, gas supply and other infrastructure. The model only needs to extract the names of such government departments and public enterprises and institutions that appear in the message to usually correctly extract the processing department. (2) The recall rate of the complaint object and the time of occurrence of the complaint problem is low. The possible reason is that the number of these two elements is small and the model lacks sufficient training samples. The reason for the small number of these two types of elements is mainly that for complaints such as road damage and road section lighting, the complaint object and the specific time of occurrence are difficult to verify and are not helpful for the handling of the problem. The complainant usually does not mention it. (3) The extraction effect of the harm factors caused by the complaint problem is poor, with low precision and recall. The reason for this phenomenon may be that there are many descriptions of the harm of the complaint problem, especially the harm to the writer of the message. The same harm may be mentioned repeatedly, and the model may extract it multiple times. For example, in the message "... causes the gas to return directly from the smoke pipe to every household. The smell is disgusting... When I get home, there is a pungent smell...", the harm of irritating oil smoke entering the home is described many times. In addition, the description of the harm caused by the complaint problem in the message often uses rhetorical techniques such as metaphors and exaggerations. When annotating, it is usually impossible to use the original text directly and needs to be modified. Therefore, when extracting such elements, the model also needs to generate modified entity elements based on the original text, which makes extraction difficult.

[0222] 3.6.3 Analysis of the impact of model weights: In order to explore the impact of model weights on the integration effect, this paper will analyze the search process of the model weight determination method based on the NSGA-II algorithm introduced in Section 3.4 when determining the optimal weight of the EG-ERRule method. Figure 7 The F1-0.8 index of the two tasks of entity feature extraction and relationship feature extraction on the validation set of the EG-ERRule method during the iteration process is shown. Figure 7 The dots in the figure represent the index value of the individual with the highest F1-0.8 index among the newly added individuals in one iteration, and the green line represents the index value of the individual with the highest F1-0.8 index found during the iteration process. When iterating to the 4th generation of the population, the optimal weight combination for the entity feature extraction task is searched, and when iterating to the 5th generation, the optimal weight combination for the relationship feature extraction task is searched. The iteration is stopped after both indicators have not increased in 10 consecutive iterations. Figure 7It can also be seen that the weight combination randomly initialized in the first iteration already has a good effect in the entity feature extraction task, while the relationship factor index has a large fluctuation in the search process. The possible reason is that the evaluation results of all base models on the quality of entity factors are similar in most of the validation set data, and therefore are less affected by the weights.

[0223] The evidence weight determination method based on the NSGA-II algorithm is as follows: In the iterative process, all weight combinations are combined on the validation set to obtain the F1-0.8 index of the entity and relationship feature extraction task: Figure 8 As shown, Figure 8 The red dots in represent the Pareto optimal weight combination. Figure 8 It can be seen that there is an obvious positive correlation between the F1-0.8 index values ​​of the two tasks, and the weight combination that performs well in the entity feature extraction task also performs well in the relationship feature extraction task. This shows that the two tasks have certain similarities, and using the data of the two tasks for simultaneous training can help improve the performance of the model on these two tasks.

[0224] The present invention selects the weight combination with the largest F1-0.8 index on the two tasks as the final weight combination, that is, the weight combination represented by the red dot in the upper right corner. The model weight of this point is shown in Table 3.6. The F1-0.8 index values ​​of the EG-ERRule method on the two tasks on the validation set under different model weights are shown in Table 3.6. Fig. 9 As shown. Fig. 9 It can be found that: (1) When the weight of the ChatGLM3 model is high, the effect of the ensemble method is generally poor. As shown in Table 3.4, the confidence mean of the other two base models on the ChatGLM3 model generation results is low. It is speculated that there is a big difference between the text generation style of the ChatGLM3 model and the label. The other two models cannot accurately evaluate the quality of the ChatGLM3 model generation results. Therefore, when the ChatGLM3 model weight is high, the ensemble result is poor. Similarly, among the model weights finally searched, the ChatGLM3 model has a low weight. (2) When the weight of a model is large, the effect of the ensemble method is inferior to the effect when the model weights are more balanced. The possible reason is that when the weight of a model is too large, other models cannot correct it when the model makes an error, and the ensemble result is closer to the result of a single model. Secondly, when the model with a large weight mistakenly gives too high a confidence to the termination token, the ensemble method will terminate the generation early, resulting in the inability to generate subsequent elements. For the same reason, the ensemble result of some weight combinations is inferior to the result of a single model.

[0225] Table 3.6 Model weights of the EG-ERRule method

[0226]

[0227] As mentioned above, since online government messages are informal texts with missing words, missing components, and other problems, traditional information extraction methods based on sequence classification or pointer networks cannot identify valid information fragments and extract clear and accurate element information. This invention proposes a corresponding solution to the problems existing in the task of extracting message information from online government platforms. The main work and conclusions of this invention are as follows:

[0228] (1) Research on the construction method of the feature extraction base model based on the fine-tuning of the large language model instruction. The text of online political question and answer messages is highly colloquial, and there is a lack of labeled data in the field of political question and answer message element extraction. It is difficult for traditional models to extract concise and clear element information from them. In response to this problem, the present invention proposes a method for constructing a feature extraction model based on the fine-tuning of the large language model instruction. The method uses a large language model to perform feature extraction tasks in a sequence generation manner, and uses the good text generation ability of the large language model to extract concise and clear element content from the colloquial element description. In the instruction fine-tuning, the method first uses information extraction data from other fields for fine-tuning to improve the general extraction ability of the model and alleviate the problem of insufficient message element extraction data; then a complaint-type political question and answer message instruction data set is built to further fine-tune the model, and the general extraction ability learned by the model is transferred to the task of extracting political question and answer messages. Experimental results show that fine-tuning with information extraction data from other fields can effectively improve the political question element extraction effect of the model.

[0229] (2) Research on factor extraction methods based on evidence theory and ensemble learning. In view of the problem of error propagation in the sequence generation process of large models, the present invention proposes a factor extraction method based on evidence theory and ensemble learning. This method uses a search strategy to search for candidate factors, calculates the conditional probability geometric mean of the candidate factors as an evaluation index, uses the NSGA-II algorithm to search for appropriate model weights, and uses the evidence reasoning rule method to fuse the evaluation results of different models, selects appropriate factors as the integration results, and generates subsequent factors based on the integration results. Experimental results show that this method can effectively improve the accuracy of generated factors, avoid the generated erroneous factors affecting the generation of subsequent factors, and improve the accuracy of factor extraction.

Claims

1. A method for extracting elements from online message text based on large language model integration, characterized in that: Used to perform the task of extracting elements from government messages, where the elements include entity elements and / or relationship elements, and the extraction task includes entity recognition task and / or relationship recognition task; The extraction method is applied to a large language model, the large language model includes at least two base models with different vocabularies, wherein the text describes the text of the element to be extracted, the instruction describes the type and output format of the element in the text of the element to be extracted, and the answer describes the extracted element output by the base model according to the instruction; The network message text element extraction method based on large language model integration includes: S100. Input a sample token sequence into the base model, wherein the sample token sequence includes an instruction, a text, and a token sequence of extracted elements; The input sample token sequences of different base models are the same, but the vocabulary of different base models are different; Wherein, if the sample token sequence does not have the extracted element, the token sequence of the extracted element is empty; S200. Any base model outputs a token sequence of k candidate elements according to the input sample token sequence; S300. Calculate the confidence of any candidate element in all candidate elements output by all base models on any base model, and calculate the confidence of the termination token; S400. Assign weights to any base model; S500. Taking the confidence of any candidate element on any base model and the confidence of the termination token as the confidence of the proposition of the identification framework of the evidence reasoning rule, and generating evidence of any base model according to the confidence of the proposition of any base model in the identification framework; S600. Calculate the weighted confidence distribution of the evidence of any base model according to the evidence of any base model and the weight of any base model; S700. According to the weighted confidence distribution of the evidence of any base model, the evidence of all base models is integrated to obtain integrated evidence, and the element with the highest probability in the integrated evidence is taken as the integration result. If the integration result is a candidate element, the candidate element is taken as an extracted element and recorded in the answer, and steps S100-S700 are iteratively performed; If the integration result is a stop token, the loop is stopped, and the extracted elements recorded in the answer are output as the extracted elements output by the large language model; The large language model is trained based on the following method: S110. Constructing a data set of instructions for extracting elements from political messages, wherein the sample of the instruction data set includes instructions, text, and answers; S120. Input the sample into the base model, and adjust the weight of the base model parameters according to the loss function; The categories of entity elements include complaint issues, the time when the complaint issues occurred, the location where the complaint issues occurred, the object of the complaint, the harm caused by the complaint issues, the appeal, the handling department and the handling measures; the relationship elements include the corresponding relationship between the handling department and the handling measures; In step S110, a data set of instructions for extracting elements from political messages is constructed, including S111. Filter and clean the data of government message; S112. Annotate the data after text filtering and data cleaning; S113. Processing the annotation data of the text according to the instruction data template base model, wherein the entity type list includes a list of entities of the category of the entity elements, and the relationship type list is execution; S114. Sort the entity extraction elements or relationship extraction element triplets of the answers of the base model, the entity extraction elements in the answers are sorted according to the order in which the entity elements appear in the text, and the relationship triplets of the relationship extraction elements in the answers are sorted based on the order in which the treatment measures appear in the text as the primary sorting basis and the order in which the treatment departments appear in the text as the secondary sorting basis; The base model in step S120 includes Embedding layer, used to convert the input token sequence into the corresponding embedding vector; Multiple Transformer decoding layers are used to capture the semantic features of the token in the input sentence and generate an embedding vector of the input token containing contextual semantic information, including feedforward networks and multi-head attention; The language model output layer is used to calculate the probability that the embedding vector containing contextual semantic information belongs to each token in the vocabulary; Among them, a linear layer with a bias term is expressed as: y=Wx+b Where: x is the input of the linear layer, is the weight of the linear layer, b is the bias term, and y is the output value; The two linear layers in the feed-forward network are adapted with adapter layers, and the linear layers adapted with adapter layers are expressed as: y=(W+BA)x+b=Wx+BAx+b x is the input of the linear layer, is the weight of the linear layer, b is the bias term, y is the output value, is the weight of one of the adapter layers, is the weight of one of the adapter layers, r represents the hyperparameter, m represents the number of base models, n represents the length of the token sequence in the token sequence, and r is much smaller than the smaller one of m and n; The elements of weight B of the adapter layer are initialized to 0, and the elements of weight A are randomly initialized according to Gaussian distribution; in: Freeze the weights W of the linear layer during base model training, and adjust the weights B and A of the adapter layer; After the base model training is completed, the weights B and A of the adapter layer are merged into the weight W of the linear layer, and the updated linear layer weight W′=W+BA; Step S120 inputs the sample into the base model and adjusts the weights of the base model parameters according to the loss function, including The token sequence of the sample of the political message element extraction instruction dataset is T = {t1, t2, ..., t n }, where n is the number of tokens in the token sequence, t i represents the i-th token, t i+1 represents the i+1th token, assuming that each t i Each corresponds to a probability distribution t′ i , t′ i The random variable of is the value of all tokens in the model vocabulary, of which only the token numbered t i The probability of the token is 1, and the probability of other tokens is 0; The token sequence T is input into the linear layer and the base model with the adapter layer. After the embedding layer, multiple Transformer decoding layers, and the language model output layer, the base model output matrix Where vocab_size is the total number of tokens in the model vocabulary, and the matrix L represents the matrix of the cosine values ​​of the embedding vector of each token in the input sentence and the standard output embedding of all tokens in the output layer of the language model; Perform softmax processing on the matrix L row by row to obtain the matrix L′, where L′ represents the conditional probability distribution of the token generated at the next position of each token in the token sequence T. The i-th row and j-th column of L′ represent the conditional probability of generating the token numbered j from the i+1-th token after the first i tokens in the token sequence T. The loss function is calculated as follows: in: The token sequence of the sample of the political message element extraction instruction dataset is T = {t1, t2, ..., t n }, where n is the length of the token sequence in the token sequence; CrossEntropyLoss represents the cross entropy loss function; L′ i Represents the conditional probability distribution of the i+1th token; q i+1 Indicates the end of the conversation; If and only if the text corresponding to the i+1th token belongs to the answer or the dialogue termination mark of the answer, q i+1 =1, otherwise q i+1 =0; The gradients of the parameters in the base model are calculated according to the loss function, and the weights of the trainable parameters in it are updated.

2. The method for extracting elements from online message text based on large language model integration according to claim 1 is characterized in that: in, The token sequence of the candidate element in step S200 is obtained based on the following method. If the candidate element is an entity element, it includes: S1-210. The base model generates a corresponding token sequence according to the input sample text sequence and any token in the base model vocabulary; S1-220. Generate a conditional probability distribution of a corresponding token sequence according to any token in the base model vocabulary, and set the conditional probability of any token as the conditional probability of the corresponding token sequence; S1-230. The k token sequences with the highest conditional probabilities are candidate token sequences. Any of the k candidate token sequences is combined with any token in the base model vocabulary to generate a corresponding token sequence, wherein the any token is the next token of the any candidate token sequence. S1-240. According to the probability distribution that any token in the base model vocabulary is the next token of any candidate token sequence, the conditional probability of the next token is multiplied by the conditional probability of any candidate token sequence to obtain the conditional probability of the corresponding token sequence; S1-250. Iterate steps S1-230 to S1-240 until the current token of any candidate token sequence is a token represented by ",", wherein any candidate token sequence is a token sequence of entity element content and its indicator, wherein the indicator includes "(" represented by the first token of any candidate token sequence and "," represented by the current token of any candidate token sequence; S1-260. After calculating the token sequence of the entity element content, generate the confidence of the token sequence of different entity element categories, and use the token sequence of the entity element category with the highest confidence as the token sequence of the entity element category after the token sequence of the entity element content, to obtain a token sequence of a candidate element and output it as a candidate entity element; S1-270. Set k-1, iterate steps S1-230 to S1-260 until k=0, the base model outputs the token sequence of the kth candidate entity element, and iterates and exits; If the candidate element is a relational element, it includes: S2-210. The base model generates a corresponding token sequence according to any token in the base model vocabulary according to the input sample text sequence; S2-220. Generate a conditional probability distribution of a corresponding token sequence according to any token in the base model vocabulary, and set the conditional probability of any token as the conditional probability of the corresponding token sequence; S2-230. The k token sequences with the highest conditional probabilities are candidate token sequences. Any of the k candidate token sequences is combined with any token in the base model vocabulary to generate a corresponding token sequence, wherein the any token is the next token of the any candidate token sequence. S2-240. According to the probability distribution that any token in the base model vocabulary is the next token of any candidate token sequence, the conditional probability of the next token is multiplied by the conditional probability of any candidate token sequence to obtain the conditional probability of the corresponding token sequence; S2-250. Iterate steps S2-230 to S2-240 until the current token of any candidate token sequence is any token in the set of tokens starting with a right bracket in the vocabulary, and any candidate token sequence is a token sequence of a relationship element and its indicator, wherein the indicator includes "(" represented by the first token of any candidate token sequence, "," between the relationship type, subject and object, and any symbol or symbol set in the set of tokens starting with a right bracket in the vocabulary represented by the current token of any candidate token sequence, to obtain a token sequence of a candidate relationship element and output it as a candidate relationship element; S2-260. Set k-1 and iterate steps S2-230 to S2-250 until k=0. The base model outputs the token sequence of the kth relationship candidate element and the iterative execution exits.

3. The method for extracting elements from online message text based on large language model integration according to claim 2 is characterized in that: in, k represents the hyperparameter of the base model specifying the number of extracted candidate elements; In step S1-220 and / or S2-220, the conditional probability of any token does not include a traditional Chinese character token; In step S1-230 and / or S2-230, any candidate token sequence among the k and any token in the base model vocabulary are used to generate a corresponding token sequence, wherein the generated corresponding token sequence does not include a traditional Chinese character token sequence; In step 1-260, the confidence of the token sequence of the entity element category is expressed by the following formula: in: PET represents the conditional probability geometric mean of the element category of the candidate element, which represents the confidence of the token sequence of the entity element category; The token sequence of the element content of the entity candidate element is Where lv is the sequence length; The token sequence of the feature category of the entity candidate feature is Where lc is the sequence length; t comma Indicates the token corresponding to ","; i represents count.

4. The method for extracting elements from online message text based on large language model integration according to claim 3 is characterized in that: If the element extracted is an entity element, then the entity candidate element θ in step S300 e The confidence in the base model m is e ), as shown by the following formula: in: Entity candidate factor θ e The token sequence of the feature name is Where lv is the sequence length; T represents the token sequence of candidate elements; Entity candidate factor θ e The token sequence of the feature category is Where lc is the sequence length; PEC represents the conditional probability of the token ","; t comma Indicates the token corresponding to the comma; TE m Represents the set of tokens starting with a right bracket in the vocabulary of base model m; PET represents the geometric mean of the conditional probabilities of the feature categories of the candidate features; PEE stands for TE m The sum of the conditional probabilities of the tokens in The set TE represents the set of tokens starting with a right bracket in the vocabulary of the base model m; te represents the token starting with a right bracket in the vocabulary of the base model m; If the element extracted is a relation element, then the relation candidate element θ in step S300 r The confidence in the base model m is r ), as shown by the following formula: in: Candidate relation factor θ r The token sequence is Where lr is the sequence length; TE m Represents the set of tokens in the vocabulary of model m that start with a right bracket; PRE represents the conditional probability of the terminal token of the candidate element token sequence; Calculate the confidence of the termination token Confidence(m,θ end ) is shown by the following formula: Confidence(m,θ end )=P(t end |T′) in: θ end It is a proposition in the identification framework that indicates the termination of continued generation; t end Indicates the end token of the candidate element token sequence; T′ represents the sample token sequence.

5. The method for extracting elements from online message text based on large language model integration according to claim 4 is characterized in that: In step S500, evidence e of base model m is generated according to the confidence of the proposition of base model m in the identification framework. m , as shown by the following formula: in: p m,θ Indicates evidence m The confidence in the proposition θ; θ means θ e or θ r , θ represents θ e It means that entity element extraction is performed, and θ represents θ r It means that the relational element extraction is executed; ∑ ρ∈Θ Confidence(m,ρ) represents the sum of the confidence of any proposition ρ among all propositions in the framework Θ on the base model m.

6. The method for extracting elements from online message text based on large language model integration according to any one of claims 1 to 5, characterized in that: The online message text is an online political inquiry message text.

7. An electronic device, comprising: One or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the method described in any one of claims 1-6.

Images