Key information extraction and mass attitude evaluation method based on large language model

Abstract

The invention provides a key information extraction and mass attitude evaluation method based on a large language model. The method comprises the following steps: collecting comment texts of the masses on a target decision, and extracting known keywords; screening a comment text related to the known keyword by utilizing a search model; further screening a comment text related to the known keyword by utilizing a semantic understanding model; rechecking the correlation between the comment texts and the known keywords by using a large language model, and filtering out the comment texts which are misjudged to be related to the known keywords to obtain a key comment set; and a multi-agent system is constructed, each agent independently evaluates the attitude of each comment text in the key comment set to the target decision, and the user agent integrates the evaluation results of all agents and outputs the evaluation results. According to the method, a multi-level screening mechanism and an attitude evaluation mechanism based on multi-agent consensus are adopted, the calculation amount in the similarity judgment process is reduced, and the attitude evaluation accuracy and the system processing efficiency are improved.

Description

Technical Field

[0001] The present invention relates to a natural language processing method, and in particular to a key information extraction and crowd attitude evaluation method based on a large language model. Background Art

[0002] With the development of the Internet, many decision makers now publish decisions that they intend to implement or have already implemented through the Internet. Internet platforms allow the public to comment on decisions. For decisions that the public pays close attention to, the amount of information in the public comments is amplified through the Internet platform, and the comments involved can constitute a large amount of comment text data. By summarizing the comment text data, we can better understand the public's views and attitudes on the target decision, so as to adjust the decision or formulate a response strategy.

[0003] Since the review text data is large in scale and has the characteristics of being scattered, and contains a lot of content that is not strongly relevant to the decision-making content, it is difficult to summarize it manually. Existing text analysis and summarization methods have high computational complexity and slow reasoning speed when processing large-scale and scattered review texts, especially in application scenarios where large amounts of data are processed in real time, and often cannot meet performance requirements. In addition, such methods usually require a large amount of high-quality annotated data for training and fine-tuning, which has high data acquisition and annotation costs, and in the case of insufficient annotated data, the model performance decreases significantly. In addition, existing technologies usually rely on a single model for judgment, which makes it difficult to fully utilize the advantages of multiple models, and cannot fully consider the strengths and limitations of different models, resulting in insufficient reliability and comprehensiveness of the judgment results. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention proposes a method for extracting key information and evaluating public attitudes based on a large language model.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] A method for extracting key information and evaluating public attitudes based on a large language model, comprising:

[0007] 1) Collect the comments of the masses on the target decision and extract the keywords of the decision as known keywords; use the search model to screen the comments related to the known keywords to form a preliminary screening text set;

[0008] 2) For the primary screening text set, the semantic understanding model is used to further screen the comment texts related to the known keywords to form the secondary screening text set;

[0009] 3) For the second-screened text set, the large language model is used to review the relevance of the comment texts with known keywords, and the comment texts that are mistakenly judged to be related to the known keywords are filtered out, and the remaining comment texts form the key comment set;

[0010] 4) Construct a multi-agent system consisting of multiple agents and a user agent. Each agent independently evaluates the attitude of each comment text in the key comment set towards the target decision, where the attitude is support, opposition or neutral. The user agent integrates the evaluation results of all agents to form the final attitude evaluation result.

[0011] Furthermore, the search model is a BM25 model, the semantic understanding model is a BERT model, and the large language model is a ChatGPT model.

[0012] Furthermore, the user agent integrates the evaluation results of all agents by using any of the following methods: weighted voting, Bayesian reasoning, and deep learning fusion. The weighted voting refers to calculating the final evaluation result of each agent according to the weight of the agent; the Bayesian reasoning refers to using the Bayesian reasoning model to take the evaluation result of each agent as input to generate the final evaluation result; the deep learning fusion refers to using the deep learning model to take the evaluation result of each agent as input to generate the final evaluation result. The deep learning model can be selected based on the experience of those skilled in the art.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) Use large language models to improve the accuracy of semantic relevance judgment: Traditional semantic relevance judgment methods (such as BERT) often face the problem of insufficient accuracy when processing complex legal texts due to the limitations of model architecture and training data. Large language models (such as GPT-4) have stronger contextual understanding and semantic analysis capabilities through large-scale pre-training and deeper architecture. By introducing a multi-agent system based on a large language model, the present invention can combine the advantages of different models and methods and make them work together to more accurately judge the semantic relevance between keywords and sentences in legal texts.

[0015] (2) Reduce data dependency and make up for the insufficiency of legal text corpora: In legal text processing scenarios, high-quality annotated data suitable for semantic relevance judgment is usually lacking, which restricts the effectiveness of traditional methods. However, large language models have been extensively pre-trained on a large number of general and specialized corpora, covering rich semantic information and able to make up for the insufficiency of legal text data. The present invention utilizes these trained large language models, without relying on large amounts of specialized annotated data, to maintain efficient semantic relevance judgment in legal text scenarios, thereby significantly reducing the system's training and data acquisition costs.

[0016] (3) Combination of deep semantic analysis and multi-level screening: Traditional word embedding or simple statistical methods are difficult to capture deep semantic relationships in complex contexts. The present invention uses a multi-level screening mechanism, first using BM25 for coarse screening, and then using BERT to perform deep image similarity calculation to ensure that the screening results are more accurate.

[0017] (4) Improve the efficiency of semantic relevance judgment: Although existing methods such as the BERT model can provide strong and accurate semantic judgment, the calculation time increases and the processing speed is slow. The present invention introduces BM25 for preliminary screening, quickly filters out texts with low relevance to keywords, effectively reduces the calculation amount of subsequent deep models, and improves the overall processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the key information extraction and crowd attitude assessment method based on a large language model of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0020] This paper proposes a comprehensive solution combining BM25, BERT, GPT and multi-agent consensus, which has the following two key points:

[0021] 1. Multi-level screening mechanism: Use the BM25 model to perform preliminary screening of large-scale texts and quickly complete the text mechanism with the highest relevance to keywords. Use the BERT model to calculate the cosine similarity of the screened texts for a second screening. Introduce the ChatGPT model to further review the semantic relevance of the texts screened by BERT, filter possible errors through context analysis and semantic discrimination, and ensure the high reliability and accuracy of the final results.

[0022] 2. Attitude evaluation mechanism based on multi-agent consensus: The present invention introduces a multi-agent consensus mechanism to perform attitude evaluation (support, opposition or neutrality). Through the integration of multi-agent opinions, the system ensures the accuracy and consistency of attitude evaluation results, overcoming the defect that a single model is prone to deviation. Two agents (AssistantAgent) work in parallel, independently judge the attitudes of keywords and sentences, and finally summarize the results through the user agent, which improves the accuracy of judgment and the processing efficiency of the system.

[0023] The present invention significantly improves the processing efficiency and judgment accuracy of the system through a multi-level screening mechanism and an attitude evaluation mechanism based on multi-agent consensus. The following is a detailed description of the technical solution of the present invention:

[0024] (1) Multi-layer screening mechanism for extracting key information

[0025] First, collect the text data of the comments of the masses on the target decision, and extract the keywords of the decision as known keywords. This embodiment collects the text data of the comments of the masses on a certain employment policy on the Internet, and selects "employment" as a known keyword. Then use the BM25 model to filter the comment text data, and the specific process is as follows. The BM25 model is a search model based on efficient word frequency statistics, which can filter out the most relevant parts of known keywords from a large amount of text, so as to subsequently simplify the processing data scale of the model, thereby improving the overall efficiency. Other search models can also be used instead of the BM25 model, such as TF-IDF, Okapi BM25 variants or modern semantic search models (such as support-based search methods) to improve the effect and efficiency of preliminary screening.

[0026] (1.1) Preliminary word segmentation and BM25 scoring: Each comment text in the comment text data is segmented, and the BM25 score of each comment text and known keywords is calculated using the BM25 model. In this embodiment, the jieba word segmentation tool is used for word segmentation processing, and the comment text is sorted into a series of vocabulary lists.

[0027] (1.2) Top 50% text screening: Sort the comment texts according to the BM25 score, and select the top 50% of the comment texts as the initial screening text set to enter the next processing stage. This process effectively filters out the comment texts with low relevance to the known keywords and reduces the data. The purpose of this is to quickly filter out the comment texts with low relevance, reduce the subsequent computing overhead, and improve the overall processing efficiency.

[0028] (2) Deep semantic similarity calculation based on BERT model

[0029] In the initial screening text set after BM25 screening, the BERT model is further used to perform deep semantic similarity calculation. The BERT model has a strong context capture capability and can accurately model the semantic relationship of complex sentences. The BERT model can also be replaced by other pre-trained language models, such as RoBERTa, ALBERT, LegalBERT or LLaMA, which may perform better in specific tasks or fields. The main goal of this stage is to perform more accurate semantic similarity calculations on the initial screening text set to ensure the consistency of the comment text and keywords in the semantic dimension. The specific process is as follows:

[0030] (2.1) Text and keyword encoding: The BERT model is used to encode each comment text and known keywords in the initial screening text set to obtain the support representation of each comment text and known keywords in the initial screening text set.

[0031] (2.2) Average Pooling: Perform average pooling on the output of the BERT model to obtain the global support representation of each comment text and known keywords in the initial screening text set.

[0032] (2.3) Cosine similarity calculation: Based on the global support representation of each comment text and known keywords in the primary screening text set, the cosine similarity between each comment text in the primary screening text set and the known keywords is calculated to evaluate their semantic similarity. The higher the cosine similarity, the closer the comment text is to the known keywords in semantics. The comment texts whose cosine similarity with the known keywords is greater than the set threshold are retained to obtain the secondary screening text set to ensure that the screened comment texts have a high degree of semantic relevance to the keywords.

[0033] (3) Relevance review based on ChatGPT model

[0034] After two rounds of screening, the ChatGPT model (or other lightweight or domain-customized large language models, such as DistilledGPT, legal-specific models, etc.) is further introduced to review the semantic relevance judgment of the second-screen text set. The ChatGPT model understands and analyzes the context of the second-screen text set. By deeply analyzing the context of each comment text and known keywords in the second-screen text set, ChatGPT can determine whether the correlation between them is true and filter out possible misjudgments to ensure the reliability of the final result. The specific process is as follows:

[0035] (3.1) Relevance review: The ChatGPT model reviews the relevance of each comment text and the keywords in the second-screen text set to determine whether they are accurately related at the semantic level and filter out possible misjudgments. The reviewed comment texts form the key comment set.

[0036] (4) Attitude evaluation based on multi-agent opinions

[0037] After completing the semantic relevance judgment, the present invention introduces a multi-agent system (Multi-Agent System) to perform attitude evaluation. In order to improve the overall processing capability of the system, the multi-agent system consists of multiple agents (AssistantAgent) and a user agent (UserProxyAgent). The two agents in the system independently analyze the screened comment texts and derive the attitude of the comment texts in the key comment set towards the target decision through their respective analysis mechanisms, i.e., support, opposition or neutrality. The specific process is as follows:

[0038] (4.1) Agent initialization: The control module initializes each agent and user agent in turn according to the predefined configuration list. Each agent loads a preset large language model (such as GPT-4) during initialization, which is used to understand and analyze the attitude in the text.

[0039] (4.2) Multi-agent collaborative evaluation: Each agent independently evaluates the attitude of the comment text in the key comment set. Specifically, each agent uses its own model and rules to analyze the text and determine whether the text's attitude towards the target decision is "supportive", "opposed" or "neutral". Each agent works independently, thus ensuring the diversity and independence of the analysis.

[0040] When computing resources are limited or simpler tasks are being processed, a single-agent architecture can be used, which only contains one agent. A single agent is responsible for speech judgment and tendency assessment, and accuracy is ensured by combining multiple judgment criteria. The processing efficiency of a single agent is relatively low, but the system structure is simpler and suitable for small-scale data or non-real-time processing scenarios.

[0041] (4.3) Opinion mechanism integration results: After the agents complete their respective attitude evaluations, the user agent integrates the evaluation results of all agents through the opinion mechanism to form the final opinion decision. If the system adopts a single-agent architecture, there is no need for integration and the output is direct.

[0042] The opinion mechanism analyzes the opinions of different agents and reaches a consensus on attitude judgment according to preset rules to ensure the accuracy and consistency of the results. The opinion mechanism can specifically adopt weighted voting, Bayesian reasoning or deep learning fusion:

[0043] Weighted voting: Different weights are assigned to the evaluation results of each agent based on its historical accuracy or current task performance, and the final conclusion is reached through weighted voting.

[0044] Bayesian reasoning: Use the Bayesian reasoning model to integrate the evaluation results of all intelligent agents, and improve the reliability of the result integration through statistics and reasoning models. This method performs better when dealing with complex and uncertain tasks.

[0045] Deep learning fusion: Use deep learning models (such as LSTM models or Transformer models) to learn the associations between different intelligent agents, fuse the final results through neural networks, and further judge the accuracy of the network.

[0046] (5) Result output

[0047] After completing the semantic relevance judgment and attitude evaluation of the text, the final selected text, semantic relevance score and attitude evaluation results are saved as formatted data (such as JSON file) and output. Each data will contain the original text, keywords, BM25 score, BERT similarity score, ChatGPT review result and attitude evaluation result (support, oppose or neutral).

[0048] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for extracting key information and evaluating public attitude based on a large language model, characterized in that: include: 1) Collect the comments of the masses on the target decision and extract the keywords of the decision as known keywords; Use the search model to filter review texts related to known keywords to form a preliminary screening text set; 2) For the primary screening text set, the semantic understanding model is used to further screen the comment texts related to the known keywords to form the secondary screening text set; 3) For the second-screened text set, the large language model is used to review the relevance of the comment texts with known keywords, and the comment texts that are mistakenly judged to be related to the known keywords are filtered out, and the remaining comment texts form the key comment set; 4) Construct a multi-agent system consisting of multiple agents and a user agent. Each agent independently evaluates the attitude of each comment text in the key comment set towards the target decision, where the attitude is support, opposition or neutral. The user agent integrates the evaluation results of all agents to form the final attitude evaluation result.

2. The method for extracting key information and evaluating public attitude based on a large language model according to claim 1 is characterized in that: In step 1), the search model is the BM25 model.

3. The method for extracting key information and evaluating public attitude based on a large language model according to claim 2 is characterized in that: In step 1), the use of the search model to filter the review texts related to the known keywords is specifically as follows: 1.1) Use the BM25 model to calculate the BM25 score of each review text and known keywords; 1.2) According to the BM25 score, select the review texts with the top m% BM25 scores to form the initial screening text set.

4. The method for extracting key information and evaluating public attitude based on a large language model according to claim 1 is characterized in that: The semantic understanding model described in step 2) is a BERT model.

5. The method for extracting key information and evaluating public attitudes based on a large language model according to claim 4 is characterized in that: Step 2) is specifically as follows: 2.1) Use the BERT model to encode each comment text and known keywords in the initial screening text set to obtain support representations of each comment text and known keywords in the initial screening text set; 2.2) Perform average pooling on the support representations of each comment text and known keywords in the initial screening text set to obtain the global support representations of each comment text and known keywords in the initial screening text set; 2.3) Based on the global support representation of each comment text and known keywords in the primary screening text set, the cosine similarity between each comment text in the primary screening text set and the known keywords is calculated; the comment texts whose cosine similarity with the known keywords is greater than the set threshold are retained, and the retained comment texts constitute the secondary screening text set.

6. The method for extracting key information and evaluating public attitude based on a large language model according to claim 1, characterized in that: The large language model in step 3) is the ChatGPT model.

7. The method for extracting key information and evaluating public attitude based on a large language model according to claim 6 is characterized in that: Step 3) is specifically: Each comment text and known keywords in the second-screen text set are input into the ChatGPT model. The task of the ChatGPT model is set to determine whether the correlation between each comment text and the known keywords is true. If the ChatGPT model determines that it is not, the corresponding comment text will be deleted, and the comment texts finally retained will constitute the key comment set.

8. The method for extracting key information and evaluating public attitude based on a large language model according to claim 1, characterized in that: Step 4) is specifically as follows: 4.1) Build a multi-agent system and initialize each agent and user agent in the multi-agent system. During the initialization process, load a large language model for each agent; 4.2) Input each comment text in the evaluation key comment set into each agent, and each agent independently evaluates whether the attitude of each comment text towards the target decision is "support", "oppose" or "neutral"; 4.3) The user agent integrates the attitude evaluation results of all agents to form the final attitude evaluation result.

Images