Public opinion monitoring method and system based on knowledge graph enhanced multi-modal large model

Abstract

The invention relates to the field of large model application, in particular to a public opinion monitoring method and system based on a knowledge graph enhanced multi-modal large model. Comprising the following steps: A, acquiring multi-modal data through Internet and social media channels; b, carrying out cleaning, duplicate removal, uniform characterization and modal alignment preprocessing operation on the collected multi-modal data; c, a public opinion monitoring knowledge base is constructed according to the obtained multi-modal information, a knowledge graph is established, and a relation path in the knowledge graph is utilized; d, enhancing the feature extraction and semantic understanding capability of the multi-modal large model by using a knowledge graph, and performing feature extraction on the preprocessed multi-modal fusion data; e, on the basis of the extracted multi-modal feature vector, public opinion analysis and discriminant scoring are carried out, and classification is carried out according to a discriminant result; f, adjusting and optimizing pre-training parameters of the multi-modal large model according to result feedback; g, performing visual display on the public opinion analysis result; according to the invention, an efficient and accurate monitoring effect can be realized.

Description

Technical Field

[0001] The present invention relates to the field of large model applications, and in particular to a public opinion monitoring method and system based on a knowledge graph-enhanced multimodal large model. Background Art

[0002] Traditional public opinion monitoring systems rely primarily on text analysis, making it difficult to effectively process multimodal data such as images, video, and audio, particularly the correlations between these multimodal elements. This limits the comprehensiveness and accuracy of public opinion analysis and monitoring. The rapid development of AI-based multimodal big model technology offers new possibilities for building more accurate and comprehensive public opinion monitoring systems. Therefore, research on a public opinion monitoring system based on knowledge graph-enhanced multimodal big models is crucial. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the existing technology, avoid the limitations and inaccuracies of public opinion analysis and monitoring, and provide a public opinion monitoring method and system based on a knowledge graph enhanced multimodal large model. The present invention adopts a multimodal public opinion knowledge base and a knowledge graph to enhance the performance of the large model, overcome the large model illusion, and achieve efficient and accurate monitoring effects.

[0004] The purpose of the present invention is to achieve the following measures: a method for public opinion monitoring based on a knowledge graph-enhanced multimodal large model, comprising the following steps:

[0005] Step A: Obtain multimodal data through the Internet and social media channels;

[0006] Step B: Clean, remove duplicates, unify representations, and perform modality alignment preprocessing on the collected multimodal data;

[0007] Step C: Build a public opinion monitoring knowledge base based on the acquired multimodal information, create a knowledge graph, and use the relationship paths in the knowledge graph to help the model perform logical reasoning;

[0008] Step D: Use the knowledge graph to enhance the feature extraction and semantic understanding capabilities of the multimodal large model, use the pre-trained multimodal large model to perform multimodal feature fusion, and perform feature extraction on the pre-processed multimodal fusion data;

[0009] Step E: Based on the extracted multimodal feature vectors, sentiment analysis, topic identification, event detection, public opinion analysis, and discrimination scoring are performed, and classification is performed according to the discrimination results;

[0010] Step F: Adjust and optimize the pre-training parameters of the multimodal large model based on the result feedback to further improve the discrimination effect and obtain public opinion event monitoring information;

[0011] Step G: Visualize the public opinion analysis results of step E.

[0012] Preferably, in step D, the multimodal public opinion information obtained in real time is preprocessed, and data of different modalities are mapped to a shared semantic space, so that similar content of different modalities is close to each other in the semantic space, realizing data fusion and cross-modal retrieval of different modalities, and inputting into the multimodal large model.

[0013] Preferably, the step D specifically includes the following steps:

[0014] Step D1: Align entities in the collected multimodal data with nodes in the knowledge graph to enhance the model’s understanding of the entities;

[0015] Step D2: Combine the entity embeddings in the knowledge graph with the word vectors of the language model, and introduce the relationship embeddings in the knowledge graph into the model to help the model understand the semantic relationships between entities;

[0016] Step D3: Combine the retrieval model and the generation model to retrieve relevant information from the knowledge graph to assist in the generation task. Graph neural networks are used to encode the knowledge graph, retrieve subgraphs related to the input, and enhance the model’s reasoning capabilities.

[0017] Step D4: Combine the representation learning of the knowledge graph with the pre-training of the language model, jointly optimize the goals of both, use the knowledge graph to correct or optimize the output after the model generates the result, and use the rules in the knowledge graph to constrain the generation of the result during the generation process.

[0018] Preferably, in step E, clustering and recognition algorithms are used to perform sentiment analysis, topic identification, event detection and public opinion analysis tasks based on the extracted multimodal feature vectors.

[0019] Preferably, in step A, the acquired multimodal data includes text, picture, audio, and video multimodal information.

[0020] Preferably, in step G, the public opinion analysis results are visualized in the form of charts and maps.

[0021] A public opinion monitoring system based on a knowledge graph enhanced multimodal large model includes a processor that executes a computer program to implement the steps of the above-mentioned public opinion monitoring method based on a knowledge graph enhanced multimodal large model.

[0022] The beneficial effects of the present invention are as follows: By utilizing the powerful feature extraction and semantic understanding capabilities of a large multimodal model, the present invention can effectively process multimodal data such as text, images, video, and audio, enabling comprehensive and accurate monitoring of public opinion. Compared with existing technologies, this technology can simultaneously process data from multiple modalities to obtain more comprehensive public opinion information; it utilizes the powerful semantic understanding capabilities of a large multimodal model to improve the accuracy of public opinion analysis, and can rapidly process and analyze massive amounts of multimodal data, enabling real-time monitoring of public opinion; and finally, it visualizes the public opinion analysis results to facilitate user understanding and decision-making.

[0023] This invention trains a specialized multimodal large model for public opinion monitoring, mining the semantic associations of text, image, audio, and video public opinion, unifying representation and encoding to achieve multimodal public opinion monitoring. It also uses a multimodal public opinion knowledge base and knowledge graph to enhance the performance of the large model, overcome the large model illusion, and achieve efficient and accurate monitoring results. Based on the scoring of monitoring results, the knowledge base and knowledge graph are optimized to improve the monitoring performance of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a system architecture diagram of the present invention;

[0025] Figure 2 Flowchart of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] Example 1: Figure 1 、 Figure 2 As shown in FIG, a method for monitoring public opinion based on a knowledge graph-enhanced multimodal large model includes the following steps:

[0028] Step A: Obtain multimodal data through the Internet and social media channels;

[0029] Step B: Clean, remove duplicates, unify representations, and perform modality alignment preprocessing on the collected multimodal data;

[0030] Step C: Build a public opinion monitoring knowledge base based on the acquired multimodal information, create a knowledge graph, and use the relationship paths in the knowledge graph to help the model perform logical reasoning;

[0031] Step D: Use the knowledge graph to enhance the feature extraction and semantic understanding capabilities of the multimodal large model, use the pre-trained multimodal large model to perform multimodal feature fusion, and perform feature extraction on the pre-processed multimodal fusion data;

[0032] Step E: Based on the extracted multimodal feature vectors, sentiment analysis, topic identification, event detection, public opinion analysis, and discrimination scoring are performed, and classification is performed according to the discrimination results;

[0033] Step F: Adjust and optimize the pre-training parameters of the multimodal large model based on the result feedback to further improve the discrimination effect and obtain public opinion event monitoring information;

[0034] Step G: Visualize the public opinion analysis results of step E.

[0035] In step A, the acquired multimodal data includes text, pictures, audio, and video multimodal information.

[0036] In step D, the multimodal public opinion information obtained in real time is preprocessed, and the data of different modalities are mapped to a shared semantic space, so that similar content of different modalities is close to each other in the semantic space, realizing data fusion and cross-modal retrieval of different modalities, and inputting them into the multimodal large model.

[0037] Using knowledge corpora and knowledge graphs to enhance the feature extraction and semantic understanding capabilities of large models helps eliminate the illusion of large models. Knowledge graphs are structured semantic networks that contain entities, attributes, and relationships between entities. They can provide rich background knowledge and logical relationships, helping models better understand context and perform reasoning.

[0038] Step D specifically includes the following steps:

[0039] Step D1: Align entities in the collected multimodal data with nodes in the knowledge graph to enhance the model’s understanding of the entities;

[0040] Utilize the relational paths in the knowledge graph to help the model perform logical reasoning; provide the model with additional common sense knowledge to make up for the deficiencies of the pre-trained model in specific fields or common sense.

[0041] Step D2: Combine the entity embeddings in the knowledge graph (such as TransE and ComplEx) with the word vectors of the language model, and introduce the relationship embeddings in the knowledge graph into the model to help the model understand the semantic relationship between entities; in the pre-training stage, incorporate the information of the knowledge graph into the loss function or training target.

[0042] Step D3: Combine the retrieval model and the generation model to retrieve relevant information from the knowledge graph to assist in the generation task. Graph neural networks are used to encode the knowledge graph, retrieve subgraphs related to the input, and enhance the model’s reasoning capabilities.

[0043] Step D4: Combine the representation learning of the knowledge graph with the pre-training of the language model, jointly optimize the goals of both, use the knowledge graph to correct or optimize the output after the model generates the result, and use the rules in the knowledge graph to constrain the generation of the result during the generation process.

[0044] Build a public opinion monitoring knowledge base, use pre-trained multimodal large models to perform multimodal feature fusion, and extract features from the pre-processed multimodal fusion data to obtain feature vectors that can uniformly represent the semantic information of multimodal data, and normalize the feature vectors.

[0045] In step E, based on the extracted multimodal feature vectors, clustering and recognition algorithms are used to perform sentiment analysis, topic identification, event detection and public opinion analysis tasks.

[0046] In step G, the public opinion analysis results are visualized in the form of charts and maps, which helps users to intuitively understand the public opinion situation.

[0047] The present invention trains a dedicated multimodal large model to realize public opinion monitoring, mines the semantic association of text, image, audio and video public opinion, unifies the representation and encoding, and realizes multimodal public opinion monitoring;

[0048] Build a public opinion monitoring knowledge base and use knowledge graph technology to enhance the large model's ability to identify, analyze, and monitor public opinion. Align entities in multimodal data with nodes in the knowledge graph to enhance the model's understanding of entities. Utilize the relationship paths in the knowledge graph to help the model perform logical reasoning, providing the model with additional common sense knowledge to compensate for the pre-trained model's shortcomings in specific fields or common sense.

[0049] For public opinion data of different modalities, the data of different modalities are mapped into a shared semantic space, so that similar content of different modalities is close in the semantic space, realizing data fusion of different modalities and cross-modal retrieval; modality alignment is achieved by maximizing the similarity of matching image-text pairs while minimizing the similarity of unmatched pairs;

[0050] Construct a multimodal public opinion monitoring corpus dataset including text, images, audio, and video, train a large public opinion monitoring model, and improve the accuracy of public opinion monitoring;

[0051] It adopts an integrated end-edge-cloud structure of end-side data collection - edge distributed computing - cloud-side training and inference - model parameter co-evolution, and lightweight deployment of updated model parameters on the cloud, so that end-side devices and edge-side devices have powerful feature extraction and semantic understanding capabilities of complex models, and comprehensively monitor public opinion information.

[0052] A public opinion monitoring system based on a knowledge graph enhanced multimodal large model includes a processor, which executes a computer program to implement the steps of the above-mentioned public opinion monitoring method based on a knowledge graph enhanced multimodal large model, and can achieve the same effect as the public opinion monitoring method based on a knowledge graph enhanced multimodal large model.

[0053] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for public opinion monitoring based on a knowledge graph-enhanced multimodal large model, characterized by: The following steps are involved: Step A: Obtain multimodal data through the Internet and social media channels; Step B: Clean, remove duplicates, unify representations, and perform modality alignment preprocessing on the collected multimodal data; Step C: Build a public opinion monitoring knowledge base based on the acquired multimodal information, create a knowledge graph, and use the relationship paths in the knowledge graph to help the model perform logical reasoning; Step D: Use the knowledge graph to enhance the feature extraction and semantic understanding capabilities of the multimodal large model, use the pre-trained multimodal large model to perform multimodal feature fusion, and perform feature extraction on the pre-processed multimodal fusion data; Step E: Based on the extracted multimodal feature vectors, sentiment analysis, topic identification, event detection, public opinion analysis, and discrimination scoring are performed, and classification is performed according to the discrimination results; Step F: Adjust and optimize the pre-training parameters of the multimodal large model based on the result feedback to further improve the discrimination effect and obtain public opinion event monitoring information; Step G: Visualize the public opinion analysis results of step E.

2. The method for public opinion monitoring based on a knowledge graph-enhanced multimodal large model according to claim 1 is characterized by: In step D, the multimodal public opinion information obtained in real time is preprocessed, and the data of different modalities are mapped to a shared semantic space, so that similar content of different modalities is close to each other in the semantic space, realizing data fusion and cross-modal retrieval of different modalities, and inputting into the multimodal large model.

3. The method for public opinion monitoring based on a knowledge graph-enhanced multimodal large model according to claim 2 is characterized by: The step D specifically includes the following steps: Step D1: Align entities in the collected multimodal data with nodes in the knowledge graph to enhance the model’s understanding of the entities; Step D2: Combine the entity embeddings in the knowledge graph with the word vectors of the language model, and introduce the relationship embeddings in the knowledge graph into the model to help the model understand the semantic relationships between entities; Step D3: Combine the retrieval model and the generation model to retrieve relevant information from the knowledge graph to assist in the generation task. Graph neural networks are used to encode the knowledge graph, retrieve subgraphs related to the input, and enhance the model’s reasoning capabilities. Step D4: Combine the representation learning of the knowledge graph with the pre-training of the language model, jointly optimize the goals of both, use the knowledge graph to correct or optimize the output after the model generates the result, and use the rules in the knowledge graph to constrain the generation of the result during the generation process.

4. The method for public opinion monitoring based on a knowledge graph-enhanced multimodal large model according to claim 1 is characterized by: In step E, based on the extracted multimodal feature vectors, clustering and recognition algorithms are used to perform sentiment analysis, topic identification, event detection and public opinion analysis tasks.

5. The method for public opinion monitoring based on a knowledge graph-enhanced multimodal large model according to claim 1 is characterized by: In step A, the acquired multimodal data includes text, pictures, audio, and video multimodal information.

6. The method for public opinion monitoring based on a knowledge graph-enhanced multimodal large model according to claim 1 is characterized by: In step G, the public opinion analysis results are visualized in the form of charts and maps.

7. A public opinion monitoring system based on a knowledge graph-enhanced multimodal large model, comprising a processor, characterized in that: The processor executes a computer program to implement the steps of the public opinion monitoring method based on a knowledge graph enhanced multimodal large model as described in any one of claims 1-6.

Images