Industrial public opinion analysis agent construction method and system based on large language model

By constructing an industry public opinion analysis agent based on a large language model, the problem of existing technology being difficult to capture the semantic correlation and timely decision-making of industry public opinion is solved, more accurate and comprehensive big data analysis is achieved, and interpretable results are provided.

CN119886304APending Publication Date: 2025-04-25GUANGZHOU UNIVERSITY

Patent Information

Application Number
CN202411950651.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-27
Publication Date
2025-04-25

AI Technical Summary

Technical Problem

The existing industry public opinion analysis methods are difficult to effectively and accurately capture the semantic relationships of industry public opinion content, and cannot provide industry public opinion decisions in a timely manner.

Method used

The industry public opinion analysis agent construction method based on large language models is adopted. By obtaining and preprocessing industry public opinion data and professional knowledge data, an industry public opinion database, professional knowledge base and relational knowledge graph are constructed, and the large model is fine-tuned to generate industry public opinion analysis Q&A pairs, and the agent is constructed for real-time analysis by combining external tools and databases.

Benefits of technology

It realizes in-depth analysis of industry public opinion information, which can more effectively and accurately explore internal semantic relationships, timely capture changing trends and hot topics, improve the accuracy and comprehensiveness of the analysis, and provide interpretable analysis results to enhance transparency and credibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119886304A_ABST
    Figure CN119886304A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of public opinion analysis, in particular to an industry public opinion analysis agent construction method and system based on a large language model.The industry public opinion analysis agent construction method based on the large language model.An industry public opinion analysis agent is constructed, industry public opinion information is deeply analyzed, and the industry public opinion analysis efficiency is improved. Compared with other industry public opinion analysis methods, the method has the advantages that the internal semantic association of industry public opinion information can be more effectively and accurately mined, the change trend and hot topics of industry public opinions are timely captured, and the accuracy and comprehensiveness of industry public opinion analysis are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of public opinion analysis, and in particular to a method and system for constructing an industry public opinion analysis intelligent agent based on a large language model. Background Art

[0002] In the rapidly developing information age, the Internet has become the main platform for people to spread and obtain various information. The sum of industry-related social attitudes, opinions and viewpoints on the Internet has formed a certain space of industry public opinion environment. Industry public opinion analysis is an indispensable part of current corporate and social management. It involves real-time, comprehensive and accurate monitoring and analysis of public opinion and emotions. At present, industry public opinion analysis mainly uses natural language processing methods of machine learning and deep learning to perform sentiment analysis, viewpoint extraction, event relationship analysis, etc. on Internet content. It cannot effectively and accurately capture the semantic correlation of industry public opinion content and provide industry public opinion decisions in a timely manner. Therefore, it is necessary to improve it. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for constructing an industry public opinion analysis intelligent agent based on a large language model. Under the premise of ensuring that the industry public opinion analysis is accurate, real-time and effective, compared with other industry public opinion analysis methods, it has strong generalization ability, can provide industry public opinion decisions in a timely manner, and provide explainable analysis results.

[0004] The present invention adopts the following technical solution:

[0005] In a first aspect, a method for constructing an industry public opinion analysis agent based on a large language model is provided, characterized in that the method comprises the following steps:

[0006] 1) Obtain industry public opinion data and industry professional knowledge data;

[0007] 2) Preprocess industry public opinion data and industry professional knowledge data to build an industry public opinion database, industry professional knowledge base, and industry public opinion relationship knowledge graph;

[0008] 3) Based on the industry public opinion database, industry professional knowledge base, industry public opinion relationship knowledge graph, corresponding public opinion analysis method and public opinion analysis task, the industry public opinion analysis question and answer pairs are produced, and some parameters of the open source base model are fine-tuned to obtain the industry public opinion analysis model;

[0009] 4) The industry public opinion analysis model is used as the brain of the intelligent agent, and the industry public opinion analysis intelligent agent is constructed by combining the external database, tool set, planning set and action set;

[0010] 5) Use the industry public opinion analysis agent to analyze the real-time industry public opinion, obtain the real-time industry public opinion analysis results and save them in the public opinion analysis result database.

[0011] As a priority, the specific acquisition method in step 1) is:

[0012] 1.1) On the computing and storage servers, use Docker cloud native to deploy distributed crawlers and distributed Redis cache to quickly crawl and store industry public opinion data and industry professional data in large quantities;

[0013] 1.2) Pull the Redis image file in the server's Docker service, run the Redis image file as a Docker container, configure the Redis configuration file, start the Redis background service, each Redis service runs in an independent container, configure the master and slave database service nodes, and build a distributed Redis cache service;

[0014] 1.3) Use multiple Docker containers to deploy Scrapy distributed crawlers. Each crawler node runs in an independent container and will not interfere with each other. The Scrapy distributed crawler stores the crawled data in the distributed Redis cache.

[0015] Prioritizes, the distributed crawler crawls various industry public opinion data in professional public opinion information, social comments, news media reports and corporate brand promotion from at least public opinion information websites, industry public opinion websites, professional public opinion monitoring websites, social media websites and news websites; the distributed crawler crawls various industry professional knowledge data in relevant industry reports, data analysis and market research industry professional knowledge from at least industry analysis websites and search engine websites.

[0016] As a priority, the data preprocessing in step 2) specifically includes the following steps:

[0017] 2.1) Use Python to write text cleaning scripts to clean industry public opinion data and industry professional knowledge data and remove irrelevant characters;

[0018] 2.2) Use the Semantic Splitter in Python’s LangChain framework to split the cleaned text data into blocks;

[0019] 2.3) Use Python to write a data standardization script to convert the segmented text data into a unified format and standard for subsequent analysis and processing;

[0020] 2.4) Use the Bge-m3 embedding model to embed semantic vectors on the standardized data to obtain the vectorized industry public opinion database IPO_DB and industry professional knowledge base IE_KB;

[0021] 2.5) Use the convert_to_graph_documents method of the LLMGraphTransformer class in the langchain_experimental.graph_transformers framework to convert the document data of the vectorized industry public opinion database IPO_DB and the industry professional knowledge base IE_KB into graph relationship data, obtain the industry public opinion relationship knowledge graph IPOR_KG, and store the industry public opinion relationship knowledge graph IPOR_KG in the neo4j graph database for preservation.

[0022] As a priority, the specific construction method of the industry public opinion analysis model in step 3) includes the following steps:

[0023] 3.1) Find the graph element relationship GER in the industry public opinion relationship knowledge graph IPOR_KG, perform one or several hops of iterative search in the knowledge graph, and search the corresponding industry public opinion data IPO_Data and industry professional knowledge data IEK_Data in the industry public opinion database IPO_DB and the industry professional knowledge base IE_KB based on the graph element relationship GER to form a formatted industry public opinion relationship knowledge triple;

[0024] 3.2) Determine the corresponding public opinion analysis method and public opinion analysis task based on the industry public opinion relationship knowledge triple IPOR_KT;

[0025] 3.3) Based on the industry public opinion relationship knowledge triple IPOR_KT, the corresponding public opinion analysis method and public opinion analysis task, the thinking chain CoT method is used to construct and generate the prompt words of the industry public opinion analysis question and answer pair IPOA_QA. The prompt words include the industry public opinion relationship knowledge triple IPOR_KT, the corresponding public opinion analysis method and public opinion analysis task description, and the output format of the public opinion analysis question and answer pair IPOA_QA;

[0026] 3.4) The prompt word is input into the GPT-4 generation model to generate an industry public opinion analysis question and answer pair IPOA_QA that meets the requirements, and the data of the industry public opinion analysis question and answer pair IPOA_QA is standardized and saved in a JSON file;

[0027] 3.5) The obtained industry public opinion analysis question and answer pair IPOA_QA is used to fine-tune some parameters of LoRA on the Llama open source base model to obtain the industry public opinion analysis model IPOA_LLM with industry public opinion analysis capabilities.

[0028] As a priority, the specific steps of constructing the industry public opinion analysis agent in step 4) are:

[0029] 4.1) Use the langchain_community.llms class in the langchain framework to load the industry public opinion analysis model IPOA_LLM as the brain of the intelligent agent;

[0030] 4.2) Based on the vectorized industry public opinion database IPO_DB, the vectorized industry professional knowledge base IE_KB and the industry public opinion relationship knowledge graph IPOR_KG, a retriever is constructed using the langchain framework;

[0031] 4.3) Use the tools class in the langchain framework to construct multiple tools and combine multiple tools into a toolset;

[0032] 4.4) Based on the original public opinion prompt, select the corresponding retrieval data, planning step description and action task description from the retriever, planning set and action set to construct a new industry public opinion analysis prompt;

[0033] 4.5) Based on the loaded industry public opinion analysis model IPOA_LLM, tools tool set and industry public opinion analysis prompt, use the create_openai_functions_agent of the agents class in the langchain framework to build the industry public opinion analysis agent IPOA_Agent.

[0034] Preferably, the tools include, but are not limited to, search engines and text mining tools.

[0035] As a preference, the specific method of step 5) comprises the following steps:

[0036] 5.1) Use the AgentExecutor method of the agents class in the langchain framework to start the industry public opinion analysis agent IPOA_Agent;

[0037] 5.2) Input the prompt word, search the external database, and with the help of the tools in the tool set, output the industry public opinion analysis results corresponding to the industry public opinion analysis action task, and save the industry public opinion analysis results in the public opinion analysis result database.

[0038] As a priority, the parameter fine-tuning methods include but are not limited to LoRA, Prefix-Tuning and Adapter;

[0039] Evaluation and analysis model methods include but are not limited to BERTScore, G-EVAL and InstructScore.

[0040] In a second aspect, a system for constructing an industry public opinion analysis agent based on a large language model is provided, which is characterized by comprising:

[0041] Data acquisition module, used to obtain industry public opinion data and industry professional knowledge data;

[0042] The data preprocessing module is used to preprocess industry public opinion data and industry professional knowledge data, and build an industry public opinion database, an industry professional knowledge base, and an industry public opinion relationship knowledge graph;

[0043] The big model building module is used to produce industry public opinion analysis question-answer pairs based on the industry public opinion database, industry professional knowledge base, industry public opinion relationship knowledge graph, corresponding public opinion analysis methods and public opinion analysis tasks, and fine-tune some parameters of the open source base big model to obtain the industry public opinion analysis big model;

[0044] The intelligent agent building module is used to use the industry public opinion analysis big model as the brain of the intelligent agent, and combine the external database, tool set, planning set and action set to build the industry public opinion analysis intelligent agent; and

[0045] The analysis result acquisition module is used to use the industry public opinion analysis intelligent agent to analyze the real-time industry public opinion, obtain the real-time industry public opinion analysis results and save them in the public opinion analysis result database.

[0046] The beneficial effects of the present invention are:

[0047] An industry public opinion analysis intelligent entity has been constructed to conduct in-depth analysis of industry public opinion information. Compared with other industry public opinion analysis methods, it can more effectively and accurately explore the intrinsic semantic correlations of industry public opinion information, timely capture the changing trends and hot topics of industry public opinion, and improve the accuracy and comprehensiveness of industry public opinion analysis.

[0048] On the premise of ensuring that industry public opinion analysis is accurate, real-time and effective, compared with other industry public opinion analysis methods, it has strong generalization capabilities, can provide industry public opinion decisions in a timely manner, and provide explainable analysis results, enhancing the transparency and credibility of industry public opinion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A flowchart of the steps of a method for constructing an industry public opinion analysis agent based on a large language model according to the present invention;

[0051] Figure 2 This is a structural diagram of a system for constructing an industry public opinion analysis agent based on a large language model according to the present invention;

[0052] Figure 3 A schematic diagram of the steps of a large model for industry public opinion analysis of a method for constructing an intelligent agent for industry public opinion analysis based on a large language model of the present invention;

[0053] Figure 4 A schematic diagram of the steps of an industry public opinion analysis intelligent agent of a method for constructing an industry public opinion analysis intelligent agent based on a large language model of the present invention;

[0054] In the figure:

[0055] 1. Data acquisition module;

[0056] 2. Data preprocessing module;

[0057] 3. Large model building module;

[0058] 4. Agent building module;

[0059] 5. Analysis result acquisition module. DETAILED DESCRIPTION

[0060] The example embodiments of the present disclosure may be modified in various ways. Therefore, various example embodiments are shown in the accompanying drawings and described in more detail. However, it will be understood that the present disclosure is not limited to specific example embodiments, but includes all modifications, equivalents and alternative forms without departing from the scope and spirit of the present disclosure.

[0061] Reference Figure 1 , Figure 2 and Figure 3 This embodiment provides a method for constructing an industry public opinion analysis agent based on a large language model, comprising the following steps:

[0062] 1) Obtain industry public opinion data and industry professional knowledge data.

[0063] Specifically, on the computing and storage servers, Docker cloud native is used to deploy distributed crawlers and distributed Redis cache to achieve fast and large-scale crawling and storage of industry public opinion data and industry professional data.

[0064] First, pull the Redis image file in the server's Docker service, run the Redis image file as a Docker container, configure the Redis configuration file, and start the Redis background service. Each Redis service runs in a separate container, configures the master and slave database service nodes, and builds a distributed Redis cache service.

[0065] Secondly, use multiple Docker containers to deploy Scrapy distributed crawlers. Each crawler node runs in an independent container and does not interfere with each other. Scrapy distributed crawlers store the crawled data in a distributed Redis cache.

[0066] Among them, distributed crawlers crawl professional public opinion information, social comments and opinions, news media reports, corporate brand promotion and other industry public opinion data from public opinion information websites, industry public opinion websites, professional public opinion monitoring websites, social media websites, news websites and other websites, and crawl relevant industry reports, data analysis, market research and other industry professional knowledge data from industry analysis websites, various search engines and other websites.

[0067] 2) Preprocess the industry public opinion data and industry professional knowledge data to build an industry public opinion database, an industry professional knowledge base and an industry public opinion relationship knowledge graph.

[0068] Specifically, data preprocessing includes cleaning, segmentation, data standardization, and text vectorization operations.

[0069] Text cleaning: Use Python to write a text cleaning script to clean the industry public opinion data and industry professional knowledge data, and remove irrelevant characters, such as punctuation marks and special characters.

[0070] Text segmentation: Use the Semantic Splitter in Python’s LangChain framework to segment the cleaned text data of articles, comments, etc.

[0071] Data standardization: Use Python to write data standardization scripts to convert the segmented text data into a unified format and standard for subsequent analysis and processing, including standardizing data dates, data source platforms, etc.

[0072] Text vectorization: Use the Bge-m3 embedding model to embed semantic vectors on the standardized data to obtain the vectorized industry public opinion database (Industry Public Opinion Datebase, IPO_DB) and industry professional knowledge base (Industry Expertise Knowledge Base, IE_KB).

[0073] Then, the convert_to_graph_documents method of the LLMGraphTransformer class in the langchain_experimental.graph_transformers framework is used to convert the document data of the vectorized industry public opinion database IPO_DB and the industry professional knowledge base IE_KB into graph relationship data to obtain the Industry Public Opinion Relationship Knowledge Graph (IPOR_KG), which includes the intrinsic relationships between events, characters, opinions, attitudes, emotions and industry professional knowledge, and the industry public opinion relationship knowledge graph IPOR_KG is stored in the neo4j graph database for preservation.

[0074] 3) Based on the industry public opinion database, the industry professional knowledge base, the industry public opinion relationship knowledge graph, the corresponding public opinion analysis method and public opinion analysis tasks, an industry public opinion analysis question and answer pair is produced, and some parameters of the open source base model are fine-tuned to obtain the industry public opinion analysis model.

[0075] Specifically:

[0076] The first step is to search for the graph element relationship (Graph ElementRelation, GER) in the industry public opinion relationship knowledge graph IPOR_KG, perform one or several hops iterative search in the knowledge graph, and search for the corresponding industry public opinion data IPO_Data and industry professional knowledge data IEK_Data in the industry public opinion database IPO_DB and the industry professional knowledge base IE_KB based on the graph element relationship GER to form an industry public opinion relationship knowledge triple (Industry Public Opinion RelationshipKnowledge Triple, IPOR_KT) in the format of (graph element relationship GER, industry public opinion data IPO_Data, industry professional knowledge data IEK_Data).

[0077] In the above, the graph element relationship GER is an SPO triple in the industry public opinion relationship knowledge graph IPOR_KG, defined as: GER = (Subject, Predicate, Object); the industry public opinion relationship knowledge triple IPOR_KT is defined as: IPOR_KT = (GER, IPO_Data, IEK_Data).

[0078] The second step is to determine the corresponding public opinion analysis methods and public opinion analysis tasks based on the industry public opinion relationship knowledge triple IPOR_KT. Among them, public opinion analysis methods include but are not limited to sentiment analysis, topic analysis, opinion analysis, influence analysis and trend analysis, etc., and public opinion analysis tasks include but are not limited to public opinion trend prediction, public opinion analysis report, event opinion analysis, etc. A set of industry public opinion relationship knowledge triple IPOR_KT may correspond to one or more public opinion analysis methods and public opinion analysis tasks.

[0079] The third step is to use the thinking chain CoT method to construct the prompt words for generating the industry public opinion analysis question and answer pair (Industry Public OpinionAnalysis Q&A, IPOA_QA) based on the industry public opinion relationship knowledge triple IPOR_KT, the corresponding public opinion analysis method and public opinion analysis task, where the prompt words for generating the industry public opinion analysis question and answer pair IPOA_QA include the industry public opinion relationship knowledge triple IPOR_KT, the corresponding public opinion analysis method and public opinion analysis task description, and the format of the public opinion analysis question and answer pair IPOA_QA to be output.

[0080] The fourth step is to input the prompt words for generating the industry public opinion analysis question and answer pair IPOA_QA into the GPT-4 generation model, generate a series of industry public opinion analysis question and answer pairs IPOA_QA that meet the requirements, standardize the data of the industry public opinion analysis question and answer pairs IPOA_QA, and save them in a JSON file. Among them, the generation models for generating the industry public opinion analysis question and answer pairs IPOA_QA include but are not limited to GPT-4, Llama, GLM4, Gemini, etc.

[0081] The fifth step is to fine-tune some parameters of LoRA on the Llama open source base model using the obtained industry public opinion analysis question and answer pair IPOA_QA to obtain the industry public opinion analysis model IPOA_LLM with industry public opinion analysis capabilities. Among them, the open source base model includes but is not limited to Llama, GLM4, Qwen2, Baichuan, etc., and some parameter fine-tuning methods include but are not limited to LoRA, Prefix-Tuning, Adapter, etc. The parameter fine-tuning process is iteratively fine-tuned in one or more steps according to the evaluation results of the evaluation analysis model. The evaluation analysis model method includes but is not limited to BERTScore, G-EVAL, and InstructScore.

[0082] 4) The industry public opinion analysis big model is used as the brain part of the intelligent agent, and a series of external databases, tool sets, planning sets and action sets are combined to build an industry public opinion analysis intelligent agent.

[0083] External database: the industry public opinion database IPO_DB, the industry professional knowledge base IE_KB and the industry public opinion relationship knowledge graph IPOR_KG.

[0084] Toolset: including but not limited to search engines, text mining tools, word cloud generators, and mind mapping tools.

[0085] Planning set: including but not limited to thought chains, multi-step reasoning, self-reflection and human feedback.

[0086] Action set: including but not limited to event context analysis, event viewpoint analysis, public opinion trend prediction, sentiment judgment and stance identification, etc.

[0087] Specifically:

[0088] The first step is to use the langchain_community.llms class in the langchain framework to load the industry public opinion analysis model IPOA_LLM as the brain part of the intelligent agent.

[0089] In the second step, based on the vectorized industry public opinion database IPO_DB, the vectorized industry professional knowledge base IE_KB and the industry public opinion relationship knowledge graph IPOR_KG, a retriever is constructed using the langchain framework so that the intelligent agent can subsequently perform retrieval augmented generation (RAG).

[0090] The third step is to use the tools class in the langchain framework to construct a series of tools, such as search engines, text mining tools, etc., and combine a series of tools into a toolset.

[0091] The fourth step is to select corresponding retrieval data, planning step description and action task description from the retriever, planning set and action set based on the original public opinion prompt words to construct new industry public opinion analysis prompt words.

[0092] The fifth step is to build the industry public opinion analysis agent IPOA_Agent using the create_openai_functions_agent of the agents class in the langchain framework based on the loaded industry public opinion analysis large model IPOA_LLM, the tools tool set and the industry public opinion analysis prompt.

[0093] 5) Use the industry public opinion analysis agent to analyze the real-time industry public opinion, obtain the real-time industry public opinion analysis results and save them in the public opinion analysis result database.

[0094] Specifically, the AgentExecutor method of the agents class in the langchain framework is used to start the industry public opinion analysis agent IPOA_Agent, input the original industry public opinion prompt, search the external database, and with the help of the tools in the toolset, output the industry public opinion analysis results corresponding to the industry public opinion analysis action task, and save the industry public opinion analysis results in the public opinion analysis result database.

[0095] Reference Figure 4 The present invention also provides an industry public opinion analysis agent construction system based on a large language model, comprising the following modules:

[0096] Data acquisition module: used to obtain industry public opinion data and industry professional knowledge data.

[0097] Data preprocessing module: used to preprocess the industry public opinion data and industry professional knowledge data, and build an industry public opinion database, an industry professional knowledge base and an industry public opinion relationship knowledge graph.

[0098] Big model building module: Based on the industry public opinion database, the industry professional knowledge base, the industry public opinion relationship knowledge graph, and the corresponding public opinion analysis method and public opinion analysis tasks, an industry public opinion analysis question and answer pair is produced, and some parameters of the open source base big model are fine-tuned to obtain the industry public opinion analysis big model.

[0099] Intelligent agent building module: The industry public opinion analysis big model is used as the brain part of the intelligent agent, and an industry public opinion analysis intelligent agent is constructed by combining a series of external databases, tool sets, planning sets and action sets.

[0100] Analysis result acquisition module: Use the industry public opinion analysis agent to analyze the real-time industry public opinion, obtain the real-time industry public opinion analysis results and save them in the public opinion analysis result database.

[0101] The subject matter disclosed above is to be considered illustrative and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the inventive concept. Therefore, to the maximum extent allowed by law, the scope is to be determined by the broadest permissible interpretation of the appended claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

1. A method for constructing an industry public opinion analysis agent based on a large language model, characterized in that: The following steps are involved: 1) Obtain industry public opinion data and industry professional knowledge data; 2) Preprocess industry public opinion data and industry professional knowledge data to build an industry public opinion database, industry professional knowledge base, and industry public opinion relationship knowledge graph; 3) Based on the industry public opinion database, industry professional knowledge base, industry public opinion relationship knowledge graph, corresponding public opinion analysis method and public opinion analysis task, the industry public opinion analysis question and answer pair is produced, and some parameters of the open source base model are fine-tuned to obtain the industry public opinion analysis model; 4) The industry public opinion analysis model is used as the brain of the intelligent agent, and the industry public opinion analysis intelligent agent is constructed by combining the external database, tool set, planning set and action set; 5) Use the industry public opinion analysis agent to analyze the real-time industry public opinion, obtain the real-time industry public opinion analysis results and save them in the public opinion analysis result database.

2. According to claim 1, a method for constructing an industry public opinion analysis agent based on a large language model is characterized in that: The specific acquisition method in step 1) is: 1.1) On the computing and storage servers, use Docker cloud native to deploy distributed crawlers and distributed Redis cache to quickly crawl and store industry public opinion data and industry professional data in large quantities; 1.2) Pull the Redis image file from the server's Docker service, run the Redis image file as a Docker container, configure the Redis configuration file, start the Redis background service, run each Redis service in an independent container, configure the master-slave database service nodes, and build a distributed Redis cache service; 1.3) Use multiple Docker containers to deploy Scrapy distributed crawlers. Each crawler node runs in an independent container and will not interfere with each other. The Scrapy distributed crawler stores the crawled data in the distributed Redis cache.

3. According to claim 2, a method for constructing an industry public opinion analysis agent based on a large language model is characterized in that: The distributed crawler crawls professional public opinion information, social comments, news media reports and various industry public opinion data from corporate brand publicity from at least public opinion information websites, industry public opinion websites, professional public opinion monitoring websites, social media websites and news websites; The distributed crawler crawls various industry expertise data in relevant industry reports, data analysis and market research industry expertise from at least industry analysis websites and search engine websites.

4. According to claim 3, a method for constructing an industry public opinion analysis agent based on a large language model is characterized in that: The data preprocessing in step 2) specifically includes the following steps: 2.1) Use Python to write text cleaning scripts to clean industry public opinion data and industry professional knowledge data and remove irrelevant characters; 2.2) Use the Semantic Splitter in Python’s LangChain framework to split the cleaned text data into blocks; 2.3) Use Python to write a data standardization script to convert the segmented text data into a unified format and standard for subsequent analysis and processing; 2.4) Use the Bge-m3 embedding model to embed semantic vectors on the standardized data to obtain the vectorized industry public opinion database IPO_DB and industry professional knowledge base IE_KB; 2.5) Use the convert_to_graph_documents method of the LLMGraphTransformer class in the langchain_experimental.graph_transformers framework to convert the document data of the vectorized industry public opinion database IPO_DB and the industry professional knowledge base IE_KB into graph relationship data, obtain the industry public opinion relationship knowledge graph IPOR_KG, and store the industry public opinion relationship knowledge graph IPOR_KG in the neo4j graph database for preservation.

5. According to the method for constructing an industry public opinion analysis agent based on a large language model according to claim 4, it is characterized in that: The specific construction method of the industry public opinion analysis model in step 3) includes the following steps: 3.1) Find the graph element relationship GER in the industry public opinion relationship knowledge graph IPOR_KG, perform one or several hops of iterative search in the knowledge graph, and search the corresponding industry public opinion data IPO_Data and industry professional knowledge data IEK_Data in the industry public opinion database IPO_DB and the industry professional knowledge base IE_KB based on the graph element relationship GER to form a formatted industry public opinion relationship knowledge triple; 3.2) Determine the corresponding public opinion analysis method and public opinion analysis task based on the industry public opinion relationship knowledge triple IPOR_KT; 3.3) Based on the industry public opinion relationship knowledge triple IPOR_KT, the corresponding public opinion analysis method and public opinion analysis task, the thinking chain CoT method is used to construct and generate the prompt words of the industry public opinion analysis question and answer pair IPOA_QA. The prompt words include the industry public opinion relationship knowledge triple IPOR_KT, the corresponding public opinion analysis method and public opinion analysis task description, and the output format of the public opinion analysis question and answer pair IPOA_QA; 3.4) The prompt word is input into the GPT-4 generation model to generate an industry public opinion analysis question and answer pair IPOA_QA that meets the requirements, and the data of the industry public opinion analysis question and answer pair IPOA_QA is standardized and saved in a JSON file; 3.5) The obtained industry public opinion analysis question and answer pair IPOA_QA is used to fine-tune some parameters of LoRA on the Llama open source base model to obtain the industry public opinion analysis model IPOA_LLM with industry public opinion analysis capabilities.

6. According to claim 5, a method for constructing an industry public opinion analysis agent based on a large language model is characterized in that: The specific steps of constructing the industry public opinion analysis agent in step 4) are: 4.1) Use the langchain_community.llms class in the langchain framework to load the industry public opinion analysis model IPOA_LLM as the brain of the intelligent agent; 4.2) Based on the vectorized industry public opinion database IPO_DB, the vectorized industry professional knowledge base IE_KB and the industry public opinion relationship knowledge graph IPOR_KG, a retriever is constructed using the langchain framework; 4.3) Use the tools class in the langchain framework to construct multiple tools and combine multiple tools into a toolset; 4.4) Based on the original public opinion prompt, select the corresponding retrieval data, planning step description and action task description from the retriever, planning set and action set to construct a new industry public opinion analysis prompt; 4.5) Based on the loaded industry public opinion analysis model IPOA_LLM, tools tool set and industry public opinion analysis prompt, use the create_openai_functions_agent of the agents class in the langchain framework to build the industry public opinion analysis agent IPOA_Agent.

7. According to claim 6, a method for constructing an industry public opinion analysis agent based on a large language model is characterized in that: Such tools include, but are not limited to, search engines and text mining tools.

8. According to claim 7, a method for constructing an industry public opinion analysis agent based on a large language model is characterized in that: The specific method of step 5) comprises the following steps: 5.1) Use the AgentExecutor method of the agents class in the langchain framework to start the industry public opinion analysis agent IPOA_Agent; 5.2) Input the original public opinion prompt, search the external database, and with the help of the tools in the tool set, output the industry public opinion analysis results corresponding to the industry public opinion analysis action task, and save the industry public opinion analysis results in the public opinion analysis result database.

9. According to claim 8, a method for constructing an industry public opinion analysis agent based on a large language model is characterized in that: The parameter fine-tuning methods include but are not limited to LoRA, Prefix-Tuning and Adapter; Evaluation and analysis model methods include but are not limited to BERTScore, G-EVAL and InstructScore.

10. A system for constructing an industry public opinion analysis agent based on a large language model, characterized in that: include: Data acquisition module, used to obtain industry public opinion data and industry professional knowledge data; The data preprocessing module is used to preprocess industry public opinion data and industry professional knowledge data, and build an industry public opinion database, an industry professional knowledge base, and an industry public opinion relationship knowledge graph; The big model building module is used to produce industry public opinion analysis question-answer pairs based on the industry public opinion database, industry professional knowledge base, industry public opinion relationship knowledge graph, corresponding public opinion analysis methods and public opinion analysis tasks, and fine-tune some parameters of the open source base big model to obtain the industry public opinion analysis big model; The intelligent agent building module is used to use the industry public opinion analysis big model as the brain of the intelligent agent, and to build the industry public opinion analysis intelligent agent by combining the external database, tool set, planning set and action set; as well as The analysis result acquisition module is used to use the industry public opinion analysis intelligent agent to analyze the real-time industry public opinion, obtain the real-time industry public opinion analysis results and save them in the public opinion analysis result database.

Citation Information

Patent Citations

  • Fine adjustment method, system and equipment based on large language model and medium

    CN117290480A

  • Large language model shaping method for deeply simulating human characters

    CN118113848A

  • Big data processing method and system based on classification algorithm

    CN118585688A

Cited By

  • Intelligent reminding method, device and equipment based on large model

    CN120196733A

  • Intelligent reminder method, device and equipment based on large model

    CN120196733B

  • Large model sensitive information filtering method based on RAG and Agent

    CN120508693A