A method for key information extraction and public attitude assessment based on a large language model

Abstract

This invention proposes a method for key information extraction and public attitude assessment based on a large language model. The method includes: collecting public comments on a target decision and extracting known keywords; using a search model to filter comment texts related to the known keywords; using a semantic understanding model to further filter comment texts related to the known keywords; using a large language model to verify the relevance of the comment texts to the known keywords, filtering out comment texts misjudged as related to the known keywords, thus obtaining a key comment set; constructing a multi-agent system, where each agent independently assesses the attitude of each comment text in the key comment set towards the target decision, and a user agent integrates the assessment results of all agents and outputs them. This invention employs a multi-level filtering mechanism and an attitude assessment mechanism based on multi-agent consensus, reducing the computational load in the similarity judgment process and improving the accuracy of attitude assessment and the system's processing efficiency.

Description

Technical Field

[0001] This invention relates to a natural language processing method, and more particularly to a method for extracting key information and assessing public attitudes based on a large language model. Background Technology

[0002] With the development of the internet, many policymakers now use it to release decisions they intend to implement or have already begun, and internet platforms allow the public to comment on these decisions. For policy announcements that are of great public interest, the sheer volume of public comments is amplified by internet platforms, resulting in large datasets of commentary text. By summarizing and analyzing this commentary data, we can better understand the public's views and attitudes towards the target decisions, thereby allowing us to adjust our decisions or formulate corresponding strategies.

[0003] Because comment text data is large in scale and scattered, containing much content not strongly relevant to decision-making, it is difficult to summarize manually. Existing text analysis and summarization methods suffer from high computational complexity and slow inference speed when processing large-scale, scattered comment texts, often failing to meet performance requirements, especially in real-time applications handling massive amounts of data. Furthermore, these methods typically require large amounts of high-quality labeled data for training and fine-tuning, resulting in high data acquisition and labeling costs, and significant performance degradation when labeled data is insufficient. Moreover, existing technologies often rely on a single model for judgment, failing to fully utilize the advantages of multiple models and comprehensively consider the strengths and limitations of different models, leading to insufficient reliability and comprehensiveness of the judgment results. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a method for extracting key information and assessing public attitudes based on a large language model.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows:

[0006] A method for key information extraction and public attitude assessment based on a large language model includes:

[0007] 1) Collect public comments on the target decision and extract the keywords of the decision as known keywords; use a search model to filter the comment texts related to the known keywords to form a preliminary text set;

[0008] 2) For the initial set of texts, a semantic understanding model is used to further filter comment texts related to known keywords, forming a second set of texts;

[0009] 3) For the second-screened text set, use a large language model to verify the relevance of the comment texts to the known keywords, filter out the comment texts that are misjudged as being related to the known keywords, and retain the comment texts to form the key comment set;

[0010] 4) Construct a multi-agent system consisting of multiple agents and a user agent. Each agent independently evaluates the attitude of each comment text in the key comment set towards the target decision, which is either supportive, opposing, or neutral. The user agent integrates the evaluation results of all agents to form the final attitude evaluation result.

[0011] Furthermore, the search model is the BM25 model, the semantic understanding model is the BERT model, and the large language model is the ChatGPT model.

[0012] Furthermore, when integrating the evaluation results of all agents, the user agent employs any one of the following methods: weighted voting, Bayesian inference, or deep learning fusion. Weighted voting refers to calculating the final evaluation result based on the evaluation result of each agent according to its weight; Bayesian inference refers to generating the final evaluation result using a Bayesian inference model with the evaluation results of each agent as input; and deep learning fusion refers to generating the final evaluation result using a deep learning model with the evaluation results of each agent as input, where the deep learning model can be selected based on the experience of those skilled in the art.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) Improving the accuracy of semantic relevance judgment using large language models: Traditional semantic relevance judgment methods (such as BERT) often face insufficient accuracy when processing complex legal texts due to limitations in model architecture and training data. Large language models (such as GPT-4), through large-scale pre-training and deeper architecture, possess stronger contextual understanding and semantic analysis capabilities. This invention introduces a multi-agent system based on large language models, which can integrate the advantages of different models and methods, enabling them to work collaboratively, thereby more accurately judging the semantic relevance between keywords and sentences in legal texts.

[0015] (2) Reducing Data Dependence and Compensating for the Shortcomings of Legal Text Corpora: In legal text processing scenarios, high-quality labeled data suitable for semantic relevance judgment is often scarce, which limits the effectiveness of traditional methods. However, large language models have been extensively pre-trained on a large number of general and specialized corpora, covering rich semantic information, which can compensate for the lack of legal text data. This invention utilizes these pre-trained large language models, without relying on a large amount of specialized labeled data, to maintain efficient semantic relevance judgment in legal text scenarios, thereby significantly reducing the training and data acquisition costs of the system.

[0016] (3). Combination of deep semantic analysis and multi-level screening: Traditional word embedding or simple statistical methods are difficult to capture deep semantic relationships in complex contexts. This invention uses a multi-level screening mechanism to first use BM25 for coarse screening and then use BERT for deep image similarity calculation to ensure more accurate screening results.

[0017] (4) Improving the efficiency of semantic relevance judgment: Existing methods such as the BERT model can provide strong and accurate semantic judgment, but the computation time is increased and the processing speed is slow. This invention introduces BM25 for preliminary screening, which quickly filters out texts with low relevance to keywords, effectively reducing the computational load of subsequent deep models and improving the overall processing efficiency. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the key information extraction and public attitude assessment method based on a large language model according to the present invention. Detailed Implementation

[0019] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0020] This invention proposes a comprehensive solution combining BM25, BERT, GPT, and multi-agent consensus, with the following two key features:

[0021] 1. Multi-layered screening mechanism: The BM25 model is used for initial screening of large-scale texts, quickly identifying the texts most relevant to the keywords. A second screening is performed using the BERT model to calculate cosine similarity. Finally, the ChatGPT model is introduced to further verify the semantic relevance of the BERT-screened texts. Through contextual analysis and semantic discrimination, potential errors are filtered out, ensuring high reliability and accuracy of the final results.

[0022] 2. Attitude Evaluation Mechanism Based on Multi-Agent Consensus: This invention introduces a multi-agent consensus mechanism for attitude evaluation (support, opposition, or neutral). By integrating the opinions of multiple agents, the system ensures the accuracy and consistency of attitude evaluation results, overcoming the bias inherent in single models. Two agents (AssistantAgent) work in parallel, independently judging the attitude of keywords and sentences, and finally, the results are aggregated through a user agent, improving the accuracy of the judgment and the system's processing efficiency.

[0023] This invention significantly improves the system's processing efficiency and judgment accuracy through a multi-level screening mechanism and an attitude evaluation mechanism based on multi-agent consensus. The following is a detailed description of the technical solution of this invention:

[0024] (1) Multi-layered screening mechanism for extracting key information

[0025] First, textual data of public comments on the target decision is collected, and keywords related to the decision are extracted as known keywords. In this embodiment, textual data of public comments on a certain employment policy on the internet are collected, and "employment" is selected as a known keyword. Then, the BM25 model is used to filter the comment text data, as follows. The BM25 model is a search model based on efficient word frequency statistics, capable of filtering out the most relevant parts of a large amount of text to the known keywords, thereby simplifying the data processing scale of the model and improving overall efficiency. Other search models can also be used to replace the BM25 model, such as TF-IDF, variants of Okapi BM25, or modern semantic search models (e.g., based on supported search methods) to improve the effectiveness and efficiency of the initial filtering.

[0026] (1.1) Preliminary word segmentation and BM25 scoring: Each comment text in the comment text data is segmented into words, and the BM25 score of each comment text with known keywords is calculated using the BM25 model. In this embodiment, the jieba word segmentation tool is used for word segmentation processing, and the comment text is sorted into a series of word lists.

[0027] (1.2) Top 50% Text Filtering: Comment texts are sorted according to their BM25 scores, and the top 50% are selected as the initial screening set for the next processing stage. This process effectively filters out comment texts with low relevance to known keywords, reducing the amount of data. The aim is to quickly filter out less relevant comment texts, reducing subsequent computational overhead and improving overall processing efficiency.

[0028] (2). Deep semantic similarity calculation based on BERT model

[0029] In the initial text set filtered by BM25, the BERT model is further used for deep semantic similarity calculation. The BERT model has powerful context-capturing capabilities, enabling it to accurately model the semantic relationships of complex sentences. The BERT model can also be replaced with other pre-trained language models, such as RoBERTa, ALBERT, LegalBERT, or LLaMA, which may perform better in specific tasks or domains. The main goal of this stage is to perform more accurate semantic similarity calculations on the initial text set, ensuring consistency between the comment text and keywords in the semantic dimension. The specific process is as follows:

[0030] (2.1) Text and keyword encoding: The BERT model is used to encode each comment text and known keywords in the initial screening text set to obtain the support representation of each comment text and known keywords in the initial screening text set.

[0031] (2.2) Mean Pooling: The output of the BERT model is subjected to mean pooling to obtain the global support representation of each comment text and known keywords in the initial screening text set.

[0032] (2.3) Cosine similarity calculation: Based on the global support representation of each comment text and known keywords in the initial screening text set, the cosine similarity between each comment text in the initial screening text set and the known keywords is calculated to evaluate their semantic similarity. The higher the cosine similarity, the closer the comment text is to the known keywords in semantics. Comment texts with a cosine similarity greater than a set threshold with the known keywords are retained to obtain the second screening text set, so as to ensure that the selected comment texts have a high semantic relevance to the keywords.

[0033] (3) Correlation verification based on the ChatGPT model

[0034] After two rounds of screening, the ChatGPT model (or other lightweight or domain-specific large language models, such as DistilledGPT, legal-specific models, etc.) is further introduced to verify the semantic relevance of the second-screened text set. By performing contextual understanding and analysis on the second-screened text set, and deeply analyzing the context of each comment text and known keywords, ChatGPT can determine whether the relevance between them is genuine and filter out possible misjudgments, ensuring the reliability of the final result. The specific process is as follows:

[0035] (3.1) Relevance Verification: The ChatGPT model verifies the relevance between each comment text and the keywords in the second-screened text set to determine whether they are semantically accurate and to filter out possible misjudgments. The verified comment texts form the key comment set.

[0036] (4) Attitude assessment based on multi-agent opinions

[0037] After determining semantic relevance, this invention introduces a multi-agent system for attitude evaluation. To improve the overall processing capacity of the system, the multi-agent system consists of multiple assistant agents and a user proxy agent. Two agents in the system independently analyze the filtered comment texts and, through their respective analysis mechanisms, determine the attitude of the comment texts in the key comment set towards the target decision—that is, support, opposition, or neutrality. The specific process is as follows:

[0038] (4.1) Agent Initialization: The control module initializes each agent and user agent sequentially according to a predefined configuration list. During initialization, each agent loads a pre-built large language model (such as GPT-4), which is used to understand and analyze attitudes in the text.

[0039] (4.2) Multi-agent collaborative evaluation: Each agent independently evaluates the attitude of the comment texts in the key comment set. Specifically, each agent uses its own model and rules to analyze the text and determine whether the text's attitude towards the target decision is "support", "oppose", or "neutral". Each agent works independently, thus ensuring the diversity and independence of the analysis.

[0040] When computing resources are limited or tasks are relatively simple, a single-agent architecture can be used, consisting of only one agent. This single agent is responsible for speech judgment and sentiment assessment, ensuring accuracy by combining multiple judgment criteria. While the processing efficiency of a single agent is relatively low, the system structure is simpler, making it suitable for small-scale data or non-real-time processing scenarios.

[0041] (4.3) Opinion Mechanism Integration Results: After the agents have completed their respective multiple attitude assessments, the user agent integrates the assessment results of all agents through the opinion mechanism to form the final opinion decision. If the system adopts a single agent architecture, no integration is required, and the results are output directly.

[0042] The opinion mechanism analyzes the opinions of different agents and reaches a consensus on attitude judgments based on preset rules, ensuring the accuracy and consistency of the results. Specifically, the opinion mechanism can employ a fusion of weighted voting, Bayesian inference, or deep learning.

[0043] Weighted voting: Based on each agent's historical accuracy or current task performance, different weights are assigned to the evaluation results of each agent, and a final conclusion is reached through weighted voting.

[0044] Bayesian inference: This approach integrates the evaluation results of all agents using a Bayesian inference model. It improves the reliability of the integrated results through statistical and inference models, and performs particularly well when dealing with complex and uncertain tasks.

[0045] Deep learning fusion: Utilizing deep learning models (such as LSTM or Transformer models) to learn the relationships between different agents, and then fusing the final results through neural networks to further determine the accuracy of the network.

[0046] (5). Output of Results

[0047] After completing the semantic relevance assessment and attitude evaluation of the texts, the final selected texts, semantic relevance scores, and attitude evaluation results are saved as formatted data (such as JSON files) and output. Each data entry will contain the original text, keywords, BM25 score, BERT similarity score, ChatGPT review results, and attitude evaluation results (support, opposition, or neutral).

[0048] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for extracting key information and assessing public attitudes based on a large language model, characterized in that, include: 1) Collect public comments on the target decision and extract the keywords of the decision as known keywords; A search model is used to filter comment texts related to known keywords, forming an initial set of texts. 2) For the initial set of texts, a semantic understanding model is used to further filter comment texts related to known keywords, forming a second set of texts; 3) For the second-screened text set, use a large language model to verify the relevance of the comment texts to the known keywords, filter out the comment texts that are misjudged as being related to the known keywords, and retain the comment texts to form the key comment set; 4) Construct a multi-agent system consisting of multiple agents and a user agent. Each agent independently evaluates the attitude of each comment text in the key comment set towards the target decision, which is either supportive, opposing, or neutral. The user agent integrates the evaluation results of all agents to form the final attitude evaluation result.

2. The method for key information extraction and public attitude assessment based on a large language model according to claim 1, characterized in that, In step 1), the search model is the BM25 model.

3. The method for extracting key information and assessing public attitudes based on a large language model according to claim 2, characterized in that, In step 1), the step of using a search model to filter comment texts related to known keywords specifically involves: 1.1) Calculate the BM25 score of each comment text relative to the known keywords using the BM25 model; 1.2) Based on the BM25 score, select the top m% of comment texts by BM25 score to form the initial screening text set.

4. The method for extracting key information and assessing public attitudes based on a large language model according to claim 1, characterized in that, The semantic understanding model mentioned in step 2) is the BERT model.

5. The method for key information extraction and public attitude assessment based on a large language model according to claim 4, characterized in that, Step 2) specifically involves: 2.1) The BERT model is used to encode each comment text and known keywords in the initial screening text set to obtain the support representation of each comment text and known keywords in the initial screening text set; 2.2) Perform average pooling on the support representations of each comment text and known keywords in the initial screening text set to obtain the global support representations of each comment text and known keywords in the initial screening text set; 2.3) Based on the global support representation of each comment text and known keywords in the initial screening text set, calculate the cosine similarity between each comment text in the initial screening text set and the known keywords; retain the comment texts with a cosine similarity greater than a set threshold with the known keywords, and the retained comment texts form the second screening text set.

6. The method for extracting key information and assessing public attitudes based on a large language model according to claim 1, characterized in that, The large language model mentioned in step 3) is the ChatGPT model.

7. The method for key information extraction and public attitude assessment based on a large language model according to claim 6, characterized in that, Step 3) specifically involves: Each comment text and known keyword in the second-screened text set is input into the ChatGPT model. The task of the ChatGPT model is to determine whether the relevance between each comment text and the known keywords is real. If the ChatGPT model determines that the relevance is not real, the corresponding comment text is deleted. The final remaining comment texts form the key comment set.

8. The method for extracting key information and assessing public attitudes based on a large language model according to claim 1, characterized in that, Step 4) specifically involves: 4.1) Construct a multi-agent system and initialize each agent and user agent in the multi-agent system. During the initialization process, load a large language model for each agent. 4.2) Input the text of each comment in the set of key comments into each agent, and each agent independently evaluates the attitude of each comment text toward the target decision as "support", "oppose" or "neutral"; 4.3) The user agent integrates the attitude evaluation results of all agents to form the final attitude evaluation result.

Images