Public opinion screening and public opinion abstract early warning method based on AI agent

Abstract

The invention relates to the technical field of artificial intelligence, data processing and information monitoring, in particular to an early warning method for public opinion screening and public opinion abstracts based on an AI agent. Multiple AI agents are used for collecting public opinion data in parallel from microblog, jittering and other multi-source channels, collection parameters are automatically adjusted according to platform interface specifications and data formats, and the data are efficiently grabbed. After collection, repeated and wrong data are removed through a cleaning algorithm, and word segmentation, part-of-speech tagging and entity recognition of the text are completed by means of tools such as Jieba word segmentation and Harbin Institute of Technology LTP. And based on deep learning models such as CNN, LSTM and the like, AI agent is trained to screen key public opinions. And for the screened public opinions, generating an abstract by using a generative AI agent of a Transform architecture. Meanwhile, public opinions are scored in multiple dimensions and weights are dynamically adjusted in combination with an emotion dictionary, machine learning and other methods. And finally, early warning thresholds such as popularity and emotional intensity are set, real-time early warning is realized, and related subjects are assisted to quickly respond to public opinions.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, big data processing, and information monitoring and analysis, and specifically provides an early warning method for public opinion screening and public opinion summary based on an AI agent. Background Art

[0002] With the popularization of the Internet, social media, news websites, various forums, etc. have become the main positions for information dissemination, generating a huge amount of public opinion data every day. Traditional public opinion processing methods face many challenges:

[0003] Low efficiency of manual processing: Relying on manual collection, screening, and analysis of public opinion data not only consumes a large amount of manpower and time, but also it is difficult to ensure the comprehensiveness and timeliness of data processing. In the face of sudden public opinion events, manual processing often cannot respond quickly, resulting in missing the best response opportunity.

[0004] Great influence of subjective factors on accuracy: Manual judgment of the emotional tendency, importance, etc. of public opinion is easily interfered by subjective factors such as personal experience, cognitive level, and emotion, making it difficult to guarantee the accuracy of public opinion analysis results.

[0005] Limited functions of existing automated systems: Current some automated public opinion monitoring systems, although they can collect data, have obvious deficiencies in intelligent analysis and precise screening. They lack the ability to understand complex semantics, cannot accurately judge the core content and potential impact of public opinion, and it is also difficult to generate a concise and clear public opinion summary, unable to provide efficient decision-making support for users. Summary of the Invention

[0006] (I) Technical problems to be solved

[0007] Aiming at the deficiencies of the existing technology, the present invention provides an early warning method for public opinion screening and public opinion summary based on an AI agent. The following problems are solved.

[0008] Improve the efficiency of public opinion processing: Solve the problem of low efficiency in processing public opinion data by traditional manual methods. Through multiple AI agents working in parallel, rapid collection, processing, and analysis of a huge amount of public opinion data are realized, meeting the real-time requirements, and ensuring timely response when public opinion events occur.

[0009] Enhance the accuracy of public opinion analysis: Overcome the problem of inaccurate public opinion analysis results caused by the influence of subjective factors in manual judgment. With the help of deep learning models and the intelligent learning ability of AI agents, accurately judge the theme, emotional tendency, and importance of public opinion data, providing a reliable basis for subsequent decision-making.

[0010] Enhance the functions of the automated system: Improve the deficiencies of the existing automated public opinion monitoring system in intelligent analysis and precise screening, enabling it to have the ability to understand complex semantics, accurately judge the core content and potential impacts of public opinion, and generate concise and clear public opinion summaries to provide users with efficient decision-making support.

[0011] (2) Technical solution

[0012] To achieve the above objectives, the present invention provides the following technical solution: A warning method for public opinion screening and public opinion summary based on an AI agent, including data collection, data processing, public opinion screening, public opinion summary generation, and a real-time warning mechanism.

[0013] Multi-source intelligent data collection: Utilize multiple AI agents with different collection strategies to simultaneously collect public opinion data from multiple sources such as social media platforms like Weibo, WeChat, Douyin, comprehensive news websites, industry vertical news websites, comprehensive forums, and professional forums. The AI agent can automatically adjust collection parameters such as request frequency and data scraping depth according to the interface specifications and data formats of different platforms to ensure efficient and accurate scraping of various types of data. After preliminary verification, the collected data is summarized into the raw data storage module to provide a data basis for subsequent processing.

[0014] Deep data preprocessing: The collected raw data enters the data preprocessing layer. The AI agent uses advanced data cleaning algorithms such as rule-based duplicate data recognition algorithms and outlier detection algorithms to remove duplicate, incorrect, and non-standard format data. For text data, natural language processing techniques are adopted. Through mature word segmentation tools (such as Jieba segmentation) for word segmentation, part-of-speech tagging tools (such as Harbin Institute of Technology LTP) for part-of-speech tagging, and named entity recognition models (such as BERT-based named entity recognition models) to identify entity information such as person names, place names, and organization names, extract key information, and store the processed data in the preprocessed data storage module.

[0015] Precise public opinion screening: Based on deep learning text classification models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their variant long short-term memory networks (LSTMs), train multiple AI agents to screen the preprocessed data. The AI agent constructs a judgment model for the theme, sentiment tendency, and importance degree of public opinion data by learning a large number of manually labeled samples. During the screening process, through weight analysis of keywords, semantic understanding, and syntactic structure parsing, for example, using word vector models (such as Word2Vec, GloVe) to convert text into vector representations and inputting them into the classification model for calculation, screen out public opinion data related to specific themes, with high heat, or strong sentiment tendency, and store it in the screened data storage module.

[0016] Intelligent public opinion summary generation: For the selected important public opinion data, a generative AI agent based on the Transformer architecture is used to generate public opinion summaries. The multi-head attention mechanism in the Transformer architecture can deeply understand and analyze public opinion texts, extract the core elements of events, such as time, place, people, event process, key viewpoints, etc. Through pre-trained language models (such as GPT series, BART, etc.), fine-tuning is carried out in combination with the characteristics of public opinion texts to generate concise and accurate summaries, which are stored in the summary data storage module.

[0017] Positive and negative scoring of network information: 1. Basic methodology: ① Sentiment dictionary method: Construct a multi-dimensional dictionary including positive word libraries, negative word libraries, degree adverb libraries, negative word libraries, and transition word libraries. Through the weighted calculation algorithm, that is, score = ∑(word weight × degree adverb coefficient × negative word coefficient) (where the negative word coefficient = (-1) 否定词数量 ), initially judge the positive and negative tendencies of public opinion information. For example, for the text "This product is not slightly useful", the calculated score is -0.5, and it is initially judged to be negative.

[0018] ② Machine learning method: Extract N-gram features (1-3 grams), perform part-of-speech tagging (focusing on analyzing adjectives and adverbs), and analyze syntactic dependency relationships to construct a feature engineering. Use the sklearn library to build models, such as building a model based on TfidfVectorizer and LinearSVC. 2. Deep learning advanced solutions: ① Fine-tuning of pre-trained models: Use the HuggingFace Transformers library, and take the BERT model as an example for fine-tuning of pre-trained models. ② Optimization of the attention mechanism in deep learning advanced solutions: Introduce the attention mechanism to enhance the model's ability to capture key emotional information, and at the same time integrate domain adaptation technology to adjust model parameters according to the characteristics of different domains to improve the model's adaptability in specific domains. 3. Multi-dimensional evaluation system: ① Fine-grained scoring model: Score public opinion information from multiple dimensions such as sentiment polarity (weight 40%), sentiment intensity (weight 25%, using a 1-5 level intensity calibration), topic relevance (weight 15%), factuality (weight 10%), and expression normality (weight 10%). ② Dynamic weight adjustment: Dynamically adjust the weights of each dimension according to different contexts, such as social media platforms or emergency events. For example, on social media platforms, the sentiment polarity weight is set to 0.5, and the factuality weight is set to 0.2; when processing information related to emergency events, the sentiment polarity weight is set to 0.3, and the factuality weight is set to 0.4.

[0019] Real-time warning mechanism: Set multiple warning thresholds, including public opinion heat threshold (such as the number of views, reposts, etc. reaching the set value within a certain time), negative sentiment intensity threshold (such as the negative sentiment score exceeding the set value), propagation speed threshold (such as the propagation range within a unit time reaching a certain level), etc. When the public opinion data reaches or exceeds the corresponding warning threshold, the AI agent automatically triggers the warning mechanism and sends warning messages to relevant personnel through multiple methods such as SMS interfaces (such as Alibaba Cloud SMS service), email sending services, instant messaging tool APIs (such as enterprise WeChat robot interfaces), etc., and at the same time attach a summary of the public opinion, so that relevant personnel can quickly grasp the public opinion situation and take effective countermeasures in a timely manner.

[0020] Compared with the prior art, the present invention provides a warning method for public opinion screening and public opinion summary based on an AI agent, which has the following beneficial effects:

[0021] (1) Efficient information acquisition: Multiple AI agents collect and process data in parallel, greatly shortening the time cycle of public opinion monitoring. From the traditional manual monitoring that may take several hours or even days to master the public opinion dynamics, it is improved to real-time or near-real-time information acquisition, ensuring that relevant entities can understand the public opinion changes in the first time. (2) Accurate analysis and decision-making: Based on the deep learning model and the intelligent analysis ability of the AI agent, it can accurately judge the emotional tendency, importance degree, etc. of public opinion, providing a reliable decision-making basis for entities such as enterprises and governments. For example, when an enterprise faces a large number of user feedbacks, it can accurately judge the advantages and problems of the product, and thus make targeted product improvements and market promotions. (3) Timely risk response: The real-time warning mechanism can immediately issue an alarm when the public opinion reaches the warning threshold, and relevant entities can quickly take measures to respond. For example, when negative public opinion breaks out in the initial stage, the enterprise intervenes in a timely manner and avoids the further deterioration of public opinion and reduces the enterprise's reputation risk and economic losses by issuing statements, improving products, etc. (4) Saving labor costs: The automated public opinion processing process reduces the dependence on a large number of manual labor and reduces labor costs. Compared with the traditional manual monitoring and analysis of public opinion that requires a large amount of manpower and time, the application of the present invention enables a small number of personnel to manage and analyze a large amount of public opinion data, improving work efficiency and resource utilization efficiency. Brief Description of the Drawings

[0022] Figure 1 It is a structural schematic diagram of a warning method for public opinion screening and public opinion summary based on an AI agent of the present invention;

[0023] Figure 2 It is a flow schematic diagram of a warning method for public opinion screening and public opinion summary based on an AI agent of the present invention; Detailed Embodiments

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0025] (1) Training of AI agent:

[0026] Collect a large amount of public opinion data, including data with different topics, sentiment tendencies, and sources, and perform manual annotation.

[0027] Use the annotated data to train the text classification model and the public opinion summary generation model, continuously adjust the model parameters, and improve the accuracy and generalization ability of the model.

[0028] (2) Data collection and preprocessing:

[0029] Start multiple AI agents, and collect public opinion data from major online platforms according to the preset collection rules.

[0030] Transmit the collected data to the preprocessing module, and the AI agent performs cleaning and preprocessing operations.

[0031] (3) Public opinion screening and summary generation:

[0032] The preprocessed data enters the public opinion screening module, and the AI agent uses the trained text classification model to screen the data.

[0033] For the important public opinions screened out, the generative AI agent generates a public opinion summary.

[0034] (4) Early warning release:

[0035] The AI agent monitors the public opinion data in real time. When the data reaches the early warning threshold, the early warning mechanism is automatically triggered, and warning information and public opinion summaries are sent to relevant personnel through various methods.

[0036] Taking the public safety field as an example, the following contents need to be considered:

[0037] Negative score: The severity of public opinion is scored on a scale of 0-10, with 10 being the most severe. For example, in a criminal offense in the public safety field, if there is a large-scale violent conflict and multiple people are injured, the negative score can be given 8 points. Because the violent conflict seriously affects social order, and multiple people being injured indicates a high degree of severity of the incident.

[0038] Event type: Select from [crime / accident / public health / social conflict / natural disaster / other]. The above-mentioned violent conflict incidents belong to the crime type. Such incidents directly violate laws and regulations and pose a serious threat to public safety.

[0039] Sensitive labels: public opinions involving [mass incidents / number of casualties / government response / rumor spread / regional sensitivity] are labeled. In this violent conflict, if there are a large number of participants, forming a mass incident, and there are a certain number of casualties, then the sensitive labels of "mass incident" and "number of casualties" are added. Mass incidents are likely to cause social panic, and the number of casualties directly reflects the serious consequences of the incident.

[0040] Credibility: Based on [Information Source Reliability / Detail Completeness / Cross-Verification], it is assessed as high / medium / low. If the public opinion information comes from authoritative media reports, the information contains detailed details such as the time, location, participants, and conflict process of the incident, and is cross-verified through multiple reliable channels, then the credibility is assessed as high. Authoritative media usually have a strict editing process, and multi-channel verification can enhance the reliability of information.

[0041] Evaluation basis: List three key reasons for judgment. For the above-mentioned violent conflict incidents, the reasons for judgment are as follows: First, the nature of the incident is serious, involving violent illegal and criminal acts; second, it has caused serious consequences of multiple injuries; third, it has been reported by authoritative media and confirmed by multiple channels, and the information is highly credible. These reasons are judged from the nature of the incident, the consequences and the reliability of the information.

[0042] Disposal suggestions: Generate three-level response suggestions [no action required / continuous observation / emergency report]. Since the incident seriously affects public safety, emergency report disposal suggestions should be given so that relevant departments can take timely measures to maintain social stability. Emergency reporting allows higher-level departments to quickly understand the situation and allocate resources for handling.

[0043] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for screening public opinion and early warning of public opinion summaries based on AI agent, characterized in that: The following steps are involved: Step 1: Multi-source intelligent data collection, using multiple AI agents with different collection strategies to collect public opinion data from social media platforms such as Weibo, WeChat, and Douyin, comprehensive news websites, industry vertical news websites, comprehensive forums, professional forums, and other multi-source channels; AI agents automatically adjust collection parameters such as request frequency and data capture depth according to the interface specifications and data formats of different platforms. After preliminary verification, the collected data is aggregated into the original data storage module; Step 2: In-depth data preprocessing. The collected raw data enters the data preprocessing layer. The AI ​​agent uses advanced data cleaning algorithms such as rule-based duplicate data recognition algorithms and outlier detection algorithms to remove duplicate, erroneous, and irregularly formatted data. For text data, natural language processing technology is used to segment words through Jieba word segmentation, use Harbin Institute of Technology LTP for part-of-speech tagging, and use the BERT-based named entity recognition model to identify entity information such as names of people, places, and institutions, extract key information, and store the processed data in the post-preprocessing data storage module. Step 3: Accurate public opinion screening step: Based on deep learning text classification models, such as convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants, long short-term memory networks (LSTM), etc., multiple AI agents are trained to screen the preprocessed data; AI agents build judgment models for the topics, sentiment tendencies and importance of public opinion data by learning a large number of manually annotated samples; in the screening process, through keyword weight analysis, semantic understanding and syntactic structure analysis, word vector models (such as Word2Vec, GloVe) are used to convert text into vector representation, which is input into the classification model for calculation, and public opinion data related to specific topics, with high popularity or strong sentiment tendencies are screened out, and stored in the screened data storage module; Step 4: Intelligent public opinion summary generation step: For the selected important public opinion data, use the generative AIagent based on the Transformer architecture to generate public opinion summaries; the multi-head attention mechanism in the Transformer architecture deeply understands and analyzes the public opinion text, extracts the core elements of the event, such as time, place, people, event process, key points, etc.; through the pre-trained language model (such as GPT series, BART, etc.), fine-tune it in combination with the characteristics of the public opinion text, generate a concise and accurate summary, and store it in the summary data storage module; Step 5: Positive and negative scoring of online information, basic methodology: construct a multi-dimensional vocabulary containing positive vocabulary, negative vocabulary, degree adverb library, negative vocabulary and transitional vocabulary, and use a weighted calculation algorithm, that is, score = ∑ (word weight × degree adverb coefficient × negative word coefficient) (where negative word coefficient = (-1) 否定词数量 ) to preliminarily judge the positive and negative tendencies of public opinion information; extract N-gram features (1-3 grammars), perform part-of-speech tagging (focusing on the analysis of adjectives and adverbs), and analyze syntactic dependencies to construct feature engineering, and use the sklearn library to build a model based on TfidfVectorizer and LinearSVC; Deep learning advanced solution: Use the HuggingFace Transformers library to fine-tune the pre-trained model using the BERT model as an example; Introduce the attention mechanism to improve the model's ability to capture key emotional information, and integrate domain adaptation technology to adjust model parameters according to the characteristics of different fields to improve the model's adaptability in specific fields; Multi-dimensional evaluation system: Score public opinion information from multiple dimensions such as emotional polarity (weight 40%), emotional intensity (weight 25%, using 1-5 level intensity calibration), topic relevance (weight 15%), factuality (weight 10%), and expression norms (weight 10%); Dynamically adjust the weights of each dimension according to different contexts, such as social media platforms or emergencies; Step 6: Real-time early warning mechanism, set multiple early warning thresholds, including public opinion heat threshold (such as the number of views and forwardings within a certain period of time reaching the set value), negative emotion intensity threshold (such as the negative emotion score exceeds the set value), propagation speed threshold (such as the propagation range within a unit of time reaches a certain degree), etc.; when the public opinion data reaches or exceeds the corresponding early warning threshold, the AI ​​agent automatically triggers the early warning mechanism and sends early warning information to relevant personnel through SMS interface (such as Alibaba Cloud SMS service), email sending service, instant messaging tool API (such as enterprise WeChat robot interface), etc., and attaches a public opinion summary.

2. The method for screening public opinion and early warning of public opinion summaries based on AI agent according to claim 1 is characterized in that: In the field of public security, a 0-10-point scale is used to negatively score the severity of public opinion. According to the type of event, it is selected from [crime / accident / public health / social conflict / natural disaster / other], and public opinion involving [mass incidents / casualties / government response / rumor spread / regional sensitivity] is marked. The credibility is assessed as high / medium / low based on [information source reliability / completeness of details / cross-validation]. Three key judgment reasons are listed as the basis for evaluation, and a three-level response recommendation of [no action required / continuous observation / urgent reporting] is generated as a disposal recommendation.

3. The method for screening public opinion and early warning of public opinion summaries based on AI agent according to claim 1 is characterized in that: The training steps of the AIagent are: collect a large amount of public opinion data, including data of different topics, sentiment tendencies and sources, and manually annotate them; use the annotated data to train the text classification model and the public opinion summary generation model, continuously adjust the model parameters, and improve the accuracy and generalization ability of the model.

Images