Multi-modal public opinion risk early warning system and method based on dynamic mapping knowledge domain and federal reinforcement learning

Abstract

The invention relates to a multi-modal public opinion risk early warning system and method based on a dynamic knowledge graph and federal reinforcement learning, and belongs to the technical field of public opinion analysis. The system comprises a data acquisition module, a modal fusion module, a knowledge graph construction module, a comparative learning module, a federal reinforcement learning modeling module and a response output module. The system is based on multi-source heterogeneous data, multi-modal semantic alignment of texts, images, videos and the like is achieved, entity relations and propagation paths are mined through a dynamically updated knowledge graph, collaborative modeling under privacy protection among terminals is achieved by fusing federal reinforcement learning, and then real-time sensing, level early warning and multi-level response strategy recommendation of public opinion risks are achieved. Based on the system, the method has the advantages of high fusion precision, high response speed and strong visual propagation path, and is widely applied to the fields of enterprise crisis management, government affair and public opinion monitoring and public safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of public opinion analysis and relates to a multimodal public opinion risk warning system and method based on dynamic knowledge graph and federated reinforcement learning. Background Art

[0002] As a key tool for real-time monitoring and analysis of speech, events and communication trends in the Internet environment, the Internet public opinion early warning system aims to identify potential risks and trigger early warning signals through intelligent means.

[0003] Current online public opinion early warning systems face significant technical bottlenecks when dealing with complex, multi-source data. Traditional methods rely on text analysis frameworks to identify risks through methods such as keyword matching and sentiment calculation. However, they lack the ability to integrate multimodal data such as images and videos, resulting in a high rate of underreporting of implicit risks associated with tampered content or sensitive scenarios. For example, while the public opinion visualization system (CN113239111A) based on knowledge graphs can construct entity relationship networks, it does not integrate non-textual information, making it difficult to cover multidimensional risk scenarios.

[0004] The update lag of static knowledge graphs further weakens the timeliness of early warnings. Existing technologies often use fixed model structures, making it difficult to update entity relationships and communication paths in a timely manner. For example, graph embedding technology (CN111241300A) relies on fixed weights for implicit association mining, making it impossible to dynamically track the path of public opinion diffusion. When negative events spread rapidly through supply chain connections, system response delays can reach hours, missing the window for intervention.

[0005] The lack of a privacy protection mechanism restricts cross-institutional collaboration. Centralized training requires sharing of raw data, which violates privacy compliance requirements and increases the risk of data leakage during collaborative modeling. While existing technologies (such as CN115934808A) suppress early warning storms, they do not introduce federated reinforcement learning or encryption mechanisms, significantly increasing the risk of customer information exposure when financial institutions collaborate. In addition, the system's automated response capabilities are weak, with over 60% of public opinion crises escalating due to manual decision-making delays exceeding three hours. For example, a brand quality issue failed to trigger an automatic response instruction, leading to the spread of the crisis.

[0006] Improvement proposals from the academic community still have limitations. Multimodal fusion models lack generalization, time series graphs (such as T-Graph) have an early warning accuracy rate of less than 80%, and the communication overhead of federated reinforcement learning frameworks is excessively high. In summary, existing technologies have yet to achieve a closed loop in multimodal semantic alignment, dynamic modeling, privacy collaboration, and intelligent response. There is an urgent need to build a real-time, accurate, compliant, and secure early warning system to overcome these bottlenecks. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a multimodal public opinion risk warning system and method based on dynamic knowledge graph and federated reinforcement learning. By integrating multimodal data dynamic perception, time series evolution modeling, privacy protection collaborative learning and intelligent response engine, it can realize the full life cycle monitoring and accurate warning of online public opinion, which is suitable for scenarios such as financial risk management, public event emergency response, and corporate brand maintenance.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning, the system includes: multimodal data collection and analysis module, cross-modal semantic fusion module, dynamic knowledge graph construction module, federated reinforcement learning modeling module, risk propagation intensity quantification module and intelligent response and visualization module, among which,

[0010] The multimodal data acquisition and analysis module obtains text data, visual data, and audio data information from social media, performs preliminary analysis on them, and preliminarily annotates each modal data to obtain an annotated multimodal dataset;

[0011] The annotated multimodal dataset is passed to the cross-modal semantic fusion module for cross-modal risk association. The cross-modal semantic fusion module aligns the semantic features of its different modal data through a comparative learning framework and jointly determines the feature risk based on the aligned features.

[0012] The dynamic knowledge graph construction module extracts entity relationships and evolution paths from the output features of the cross-modal semantic fusion module based on a temporal graph neural network to build a knowledge graph, and continuously updates the knowledge graph to reflect the development of public opinion events;

[0013] The federated reinforcement learning modeling module is based on a locally trained deep reinforcement learning model and takes the event node feature data processed by the dynamic knowledge graph construction module as input. The central server aggregates model parameters through the FedAvg algorithm protected by differential privacy and secure multi-party computing, and outputs event evolution trends and risk level prediction parameters.

[0014] The risk diffusion intensity quantification module is based on the event evolution path graph, combined with multi-hop path analysis and graph embedding technology. It obtains the diffusion probability and risk diffusion path intensity indicators from the entity relationship links and diffusion path node information in the knowledge graph to evaluate the potential risk diffusion capacity.

[0015] The intelligent response and visualization module receives risk level prediction parameters and propagation intensity indicators, generates event evolution diagrams, time evolution response diagrams and risk control panels, and realizes the automatic triggering of risk identification, monitoring and response.

[0016] Furthermore, the multimodal data acquisition and analysis module includes a text data processing submodule, a visual data processing submodule, and an audio data processing submodule. The text data processing submodule, the image data processing submodule, and the audio data processing submodule are respectively provided with corresponding complete data processing flows, each including four steps: data acquisition, data preprocessing, feature extraction, and data annotation.

[0017] The text data processing submodule collects text information related to public opinion, performs preprocessing operations such as denoising and word segmentation, and uses the pre-trained BERT model to extract keywords, sentiment polarity, and subject entities from text features, and annotates them based on timestamps and event tags.

[0018] The image data processing submodule collects multi-source image or video data, uses the YOLOv5 model to identify sensitive targets and scene information in the image, extracts target categories, positional relationships, and image emotions from image features, and annotates them with time and event semantic information.

[0019] The audio data processing submodule collects speech audio data, uses ASR technology to transcribe it into text, combines voiceprint analysis and emotion classifier to extract intonation features, speaker identity and emotional polarity in the audio features, and completes labeling based on event trigger time and content features.

[0020] Furthermore, the cross-modal semantic fusion module maps text data, image data, and audio data into a unified semantic space based on the CLIP model, where:

[0021] Set up a text encoder for text data and use the RoBERTa model to generate text vectors;

[0022] Design an image encoder for image data and use ResNet-50 to output feature vector representation of the same dimension;

[0023] An audio encoder is set up for audio data, and a recurrent neural network (CNN) is used to map the audio feature vector to a semantic space with the same dimension as the text and image data to generate an audio vector.

[0024] Optimize its semantic alignment through contrast loss function:

[0025]

[0026] Wherein, the subscript i represents the current sample, j represents the comparison sample; i ′、v i ' and a i ′The aligned feature vector of the i-th text, image and audio sample; s(t i ',v i') represents the cosine similarity between text and image features, which measures semantic consistency; s(t i ',a i ') represents the cosine similarity between text and audio features, which measures semantic consistency; τ is a temperature hyperparameter used to control the concentration of feature distribution; An exponential scaling term for text-image and text-audio feature similarity, used to calculate the comparison weights for heterogeneous samples.

[0027] Furthermore, the process of building and updating the knowledge graph by the dynamic knowledge graph construction module is as follows:

[0028] Entity node representation update unit: fuses timestamp features with entity context information to update node embedding representation;

[0029] Relationship edge weight adjustment unit: uses the attention mechanism to perform temporal weighting on the edge weights in the same event chain to improve the accuracy of propagation path modeling;

[0030] Event chain generation unit: Constructs an event graph based on the risk topic clustering results and updates the propagation path.

[0031] Furthermore, in terms of node updates, the gated recurrent unit GRU and the graph convolution operation GCN are used to collaboratively update the node representation, which is specifically achieved through the following formula:

[0032]

[0033] in, represents the hidden state of node v at time t; represents the hidden state of node v at the last moment t-1; N(v) is the set of neighbor nodes of node v; |N(v)| is the number of neighbor nodes; W e Represents the learnable graph convolution weight matrix; W x Represents feature projection; Represents multimodal features; the update mechanism output by the cross-modal semantic fusion module is used to dynamically update the representation of nodes in the knowledge graph to reflect changes in public opinion-related information;

[0034] The event chain connects event nodes in series according to the "fermentation-diffusion-dissipation" stage, and uses the LSTM model to predict the popularity trend. The formula is:

[0035] y t+k =LSTM(y t ,y t-1 ,...,y t-m+1 )

[0036] where y tRepresents the event heat value at time t, m represents the length of the historical window, which determines the number of historical time steps used for prediction. When a threshold value θ is exceeded, a predictive warning is triggered, where Δt represents the time interval and θ represents the threshold value dynamically adjusted based on industry characteristics. This warning provides an early signal for the system's subsequent risk response, helping the entire warning system respond to public opinion risks more promptly. It also collaborates with warnings from other subsequent modules to enhance the system's responsiveness to public opinion risks.

[0037] Furthermore, it is characterized in that the federated reinforcement learning warning module is composed of a local training unit, a parameter aggregation unit and a gradient compression unit, wherein:

[0038] Local training unit: Each participant trains a deep Q network based on a locally stored multimodal public opinion dataset. During training, the state-action-reward-next-state quadruple (s, a, r, s') is extracted in batches from the local experience replay buffer, and the mean square error loss function L is used. DQN For strategy optimization, the formula is:

[0039] L DQN =E(s,a,r,s′)~D[(r+γ×max a′ Q(s′,a′;θ - )-Q(s,a;θ)) 2 ]

[0040] where E(s,a,r,s′)~D[·] represents the expected operation on the quadruple (s,a,r,s′) in the experience replay buffer D; γ is the discount factor; θ is the online network parameter, θ - is the target network parameter (periodically copied and updated from the online network); s represents the current state, which contains multimodal feature information related to public opinion; a represents the action taken; r represents the reward obtained after performing action a; s' represents the next state after performing action a; Q(s, a; θ) represents the Q value determined by the online network parameter θ when performing action a in state s, that is, the estimated future cumulative reward for this action. The gradient information generated by training is transmitted to the parameter aggregation unit;

[0041] Parameter aggregation unit: The central server receives the gradient information uploaded by each participant and uses the FedAvg algorithm to calculate the weighted average for aggregation, where the weight is related to the amount of data from the participant. Gaussian noise with a variance of σ is injected into the aggregated gradient to meet the differential privacy constraint. The specific aggregation formula is:

[0042]

[0043] where θ global represents the global model parameters after aggregation; N is the number of participants; |Di | represents the dataset size of the i-th participant; |D| represents the total size of the datasets of all participants; θ i represents the model parameters obtained by local training of the i-th participant; N((0,σ 2 I)) means the mean is 0 and the variance is σ 2 I is the Gaussian noise, where I is the identity matrix and σ is used to control the noise intensity. The aggregated gradient is sent to the gradient compression unit;

[0044] Gradient compression unit: After receiving the aggregated gradient from the parameter aggregation unit, it uses Top-K sparsification technology to retain the gradient parameters of the first K proportions, compress the communication data volume, and output the compressed sparse gradient matrix for federated model updates and iterations. Under the federated reinforcement learning framework, it achieves continuous optimization and training of the multimodal public opinion risk warning model.

[0045] Furthermore, the risk propagation quantification module quantifies the risk propagation intensity of explicit and implicitly associated entities in the following ways:

[0046] Risk propagation quantification module: Calculates risk propagation intensity based on the data obtained by the multimodal data acquisition and analysis module and the structural information of the knowledge graph;

[0047] The explicit propagation strength is calculated by the following formula:

[0048]

[0049] where w k The relationship between entities is set based on their nature, including supply chain relationships and cooperative relationships, to reflect the importance of different relationships in risk transmission; is the ratio of the transaction amount between entity i and j under relationship k to the total transaction amount of entity i, extracted by multimodal data analysis, θ k Determined through historical public opinion and knowledge graph feature learning, reflecting the difficulty of risk propagation under relationship k;

[0050] The implicit propagation strength is determined by the following formula:

[0051] S 隐式 =sim(V i ,V j )×λ(t)

[0052] Where V i ,V j is the entity embedding vector generated by Node2Vec based on the knowledge graph structure and multimodal data; sim(·) is the cosine similarity function; the initial value of λ(t) is set according to the characteristics of the industry and adjusted as the public opinion evolves; the dynamic weight coefficients α(t) and β(t) are adaptively adjusted according to the stage of public opinion evolution. The formula is as follows:

[0053]

[0054] β(t)=1-α(t)

[0055] Among them, k is determined based on the historical data of industry public opinion, and t0 is the time when the public opinion breaks out; in the early stage of public opinion, β(t) dominates the exploration of potential correlations; in the later stage, α(t) dominates the accurate reflection of the actual transmission path, realizing the comprehensive quantification of the intensity of risk transmission.

[0056] Furthermore, the intelligent response and visualization module supports the generation of propagation topology maps, the display of timeline evolution diagrams, and the triggering of hierarchical corresponding strategies. Among them, the generation of propagation topology maps refers to the dynamic drawing of event propagation networks by combining entity embedding vectors and edge weights. The node color reflects the risk level, and the edge transparency represents the propagation intensity. The display of timeline evolution diagrams refers to the marking of key event nodes and the predicted heat trend. The dotted line represents the future development curve predicted by the LSTM model. The triggering of hierarchical response strategies refers to the automatic adoption of corresponding measures for risk events of different levels.

[0057] On the other hand, this paper proposes an early warning method based on the aforementioned multimodal public opinion risk early warning system based on dynamic knowledge graph and federated reinforcement learning, which includes:

[0058] The multimodal data collection and analysis module obtains text, image, and audio data information from social media, generates a multimodal dataset through preliminary analysis and annotation, and passes it to the cross-modal semantic fusion module.

[0059] The cross-modal semantic fusion module aligns the semantic features of different modalities through comparative learning, jointly judges the feature risks, and outputs the results to the dynamic knowledge graph construction module.

[0060] The dynamic knowledge graph construction module, based on a time-series graph neural network, extracts entity relationships and evolution paths from input features to build a knowledge graph and continuously updates it. The output dynamic knowledge graph information (including node hidden states, event chain heat, etc.) is passed to the federated reinforcement learning modeling module.

[0061] In the federated reinforcement learning modeling module, each data participant locally trains a deep reinforcement learning model based on input information, using graph features as state input, defining an action space, and receiving reward signals based on the effectiveness of responses. The central server aggregates parameters using the FedAvg algorithm, protected by differential privacy and secure multi-party computation, and outputs optimized risk level predictions and early warning strategies, which are then transmitted to the intelligent response and visualization module. Training data is also fed back to assist in optimizing the dynamic knowledge graph construction module.

[0062] The risk propagation intensity quantification module combines multi-hop path analysis and graph embedding technology to evaluate the explicit and implicit strength of event propagation paths based on dynamic knowledge graphs and risk level prediction information, and quantify the risk diffusion capacity. The results are used by the intelligent response and visualization modules.

[0063] The risk response and visualization display module receives risk level predictions, early warning strategies, and risk propagation intensity indicators, provides propagation topology maps, etc., triggers automated operations according to risk levels, and feeds back response results to the federated reinforcement learning modeling module to optimize strategies.

[0064] The beneficial effects of the present invention are:

[0065] This invention significantly improves the coverage and response efficiency of public opinion risk identification by integrating multimodal data with dynamic knowledge graph modeling. Compared to traditional technologies that rely on single-text analysis, the system integrates multiple sources of information, including text, images, and video, effectively identifying tampered content and sensitive scenarios, significantly reducing the rate of missed reports. For example, in financial fraud scenarios, the system accurately identifies potential risks by correlating images of tampered contracts with user complaint text, avoiding misjudgments caused by the limitations of single-modal analysis.

[0066] The dynamic knowledge graph's real-time update mechanism further optimizes the timeliness of early warnings. Traditional static systems often delay critical information due to update lags. This system, however, uses a temporal graph neural network (T-GNN) to continuously track the evolution of public opinion, rapidly capturing changes in transmission paths and significantly shortening early warning response times. For example, when negative public opinion spreads within a company's supply chain, the system dynamically updates associated nodes, generating risk transmission links in record time and providing decision makers with a critical window for intervention.

[0067] Regarding privacy, the system utilizes a federated reinforcement learning framework and privacy-enhancing technologies to ensure data security during cross-institutional collaboration. Through encrypted communication and noise injection mechanisms, sensitive information remains protected throughout the joint modeling process, eliminating the risk of raw data leakage. For example, when multinational financial institutions jointly train models, customer data is encrypted throughout the process, and the central server only receives desensitized parameters, fully complying with international privacy regulations.

[0068] Furthermore, the system's built-in hierarchical response strategy significantly improves crisis management efficiency. Traditional manual decision-making processes often lead to delays due to hierarchical approval processes. This system automatically triggers predefined actions based on risk levels, significantly reducing intervention time. For example, in the event of a brand product recall, the system automatically links public opinion data with supply chain information, immediately suspends the circulation of the affected product, and generates a public relations response plan, significantly reducing the cost and time loss of manual intervention. Through multi-dimensional technological innovation, this system has achieved systematic breakthroughs in risk identification, dynamic modeling, privacy protection, and response efficiency.

[0069] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0071] Figure 1 This is a schematic diagram of the overall structure of a multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning according to an embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram of the data processing flow of the dynamic knowledge graph construction module under an embodiment of the present invention;

[0073] Figure 3 The figure is a schematic diagram of the data processing flow of the federated reinforcement learning early warning module under an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0075] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0076] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0077] See also Figures 1 to 3 , which is a multimodal public opinion risk warning system and method based on dynamic knowledge graph and federated reinforcement learning.

[0078] Example 1

[0079] This embodiment first provides a multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning, such as Figure 1 As shown, it includes: multimodal data acquisition and parsing module, cross-modal semantic fusion module, dynamic knowledge graph construction module, federated reinforcement learning warning module, risk propagation intensity quantification module and intelligent response and visualization module, among which,

[0080] Multimodal Data Collection and Parsing Module: Responsible for collecting text, images, audio, and video data from multiple channels, including social media, e-commerce platforms, news media, and surveillance systems. After collection, the text is deduplicated, invalid characters are removed, and word segmentation is performed. Images are denoised, cropped, and normalized. Audio is denoised, standardized, and format converted. Video keyframes are extracted, and parameters are checked and adjusted. For sentiment recognition analysis, deep learning models are used to determine sentiment and score text. Image features are extracted to determine sentiment categories. Audio acoustic features are extracted to identify emotional states. Videos are combined with image, audio, and semantic information to determine sentiment. This data is ultimately organized into a structured dataset to provide input for subsequent modules.

[0081] Cross-modal semantic fusion module: Based on a contrastive learning framework, it uses a shared embedding space to align features across text, images, and speech modalities. It also enhances the model's robustness to noisy data through adversarial sample generation technology, thereby enhancing semantic consistency.

[0082] Dynamic knowledge graph construction module: Based on the temporal graph neural network (T-GNN), it extracts entity relationships and evolution paths, receives the output features of the cross-modal semantic fusion module in real time through an event trigger mechanism, and continuously updates the knowledge graph to reflect the development of public opinion events;

[0083] Federated reinforcement learning modeling module: Each data participant trains a deep reinforcement learning model (DQN) locally. The central server aggregates model parameters using the FedAvg algorithm protected by differential privacy and secure multi-party computation (MPC), and outputs risk level prediction and early warning strategies.

[0084] Risk propagation intensity quantification module: Combining multi-hop path analysis and graph embedding technology, it evaluates the explicit and implicit strength of event propagation paths and quantifies the overall risk diffusion capacity;

[0085] Intelligent response and visualization module: Provides propagation topology diagrams, time evolution diagrams, and response control panels, supporting automated operations such as freezing accounts, removing products from shelves, and generating public opinion response plans based on risk levels.

[0086] In this embodiment, the multimodal data acquisition and analysis module plays a key role. It is responsible for real-time capture of text, images, video, and audio data from social media, news platforms, and short video applications, and generates standardized data streams through a series of operations such as deduplication, noise filtering, and format conversion, providing a high-quality data foundation for subsequent modules.

[0087] The specific operation process is as follows: In terms of text parsing, the pre-trained BERT model is used to accurately extract keywords and sentiment polarity. When the sentiment score is lower than -0.7 or higher than 0.7, it will be marked as a high-risk label to highlight the risk information contained in the text. Image analysis uses a combination of YOLOv5 target detection OCR technology to identify abnormal text or sensitive scenes in tampered images. The confidence threshold is set to 0.7 to achieve a good balance between accuracy and false alarm rate to ensure the reliability of risk identification. Video processing uses key frame extraction and background soundtrack analysis technology to keenly capture risk signals in dynamic content, such as group gatherings in live broadcasts or panicky tones in audio.

[0088] The multimodal data acquisition and analysis module consists of the following submodules:

[0089] Text crawling and cleaning: This function supports the collection of text data from multiple sources, including Weibo, forums, and news platforms. After collection, advanced denoising algorithms are used to remove noise from the text. Efficient word segmentation tools are then used to perform word segmentation, converting the text into a form that facilitates subsequent analysis, laying the foundation for in-depth mining of text information.

[0090] Image and Video Parser: Leveraging cutting-edge image recognition and frame-level feature extraction technologies, it conducts detailed analysis of image and video content. It accurately identifies keywords related to people, places, and scenes in images and extracts key features from each frame of a video, facilitating understanding of the information contained in images and videos and assessing risks.

[0091] Audio Transcription and Emotion Recognition: Using ASR automatic speech recognition technology, combined with multiple emotion classifiers, we achieve structured transcription and emotion determination for audio content in live broadcasts and podcasts. This converts audio content into text and accurately identifies the underlying emotions, enabling the system to effectively process and analyze the audio information.

[0092] In this embodiment, the cross-modal semantic fusion module uses the SimCLR contrastive learning framework to enhance the cross-modal semantic alignment capability by constructing positive and negative sample pairs, and introduces a gating mechanism to balance the multimodal feature weights. The contrastive learning framework solves the semantic gap problem of heterogeneous data. Based on the CLIP (Contrastive Language-Image Pre-training) model, text and image data are mapped to a unified semantic space. The text encoder uses RoBERTa to generate a 384-dimensional vector, and the image encoder uses ResNet-50 to output feature representations of the same dimension, and optimizes their semantic alignment through the contrastive loss function:

[0093]

[0094] Wherein, the subscript i represents the current sample, j represents the comparison sample; i ′、v i ' and a i ′The aligned feature vector of the i-th text, image and audio sample; s(t i ',v i ') represents the cosine similarity between text and image features, which measures semantic consistency; s(t i ',a i ') represents the cosine similarity between text and audio features, which measures semantic consistency; τ is a temperature hyperparameter used to control the concentration of feature distribution; An exponential scaling term for text-image and text-audio feature similarity, used to calculate the comparison weights for heterogeneous samples.

[0095] In this embodiment, if Figure 2 As shown in the figure, the graph neural network structure in the dynamic knowledge graph building module includes:

[0096] Entity node representation update unit: fuses timestamp features with entity context information to update node embedding representation;

[0097] Relationship edge weight adjustment unit: uses the attention mechanism to perform temporal weighting on the edge weights in the same event chain to improve the accuracy of propagation path modeling;

[0098] Event chain generation unit: Constructs an event graph based on the risk topic clustering results and updates the propagation path.

[0099] The node representation is updated collaboratively using the gated recurrent unit (GRU) and graph convolution operation (GCN), which is specifically achieved through the following formula:

[0100]

[0101] in, represents the hidden state of node v at time t, which comprehensively reflects the current risk characteristics of the node; is the hidden state of node v at the previous moment t-1, carrying historical risk information. N(v) is the set of neighbor nodes of node v, which are connected to node v through edge relationships in the knowledge graph; |N(v)| is the number of neighbor nodes, which is used to normalize the neighbor node information to ensure that the influence of each neighbor node is reasonable. The learnable graph convolution weight matrix W e And the feature projection matrix W x , which are responsible for aggregating neighbor node information and fusing the multimodal features of the current moment, Output from the cross-modal semantic fusion module, it covers multiple aspects of information such as text, images, and audio. This formula enables nodes to dynamically update their own risk representations based on their own historical status, neighbor node information, and current multimodal features. The performance indicators are node update delay ≤ 30 seconds and event chain generation throughput ≥ 1000 items / minute, which is 90% more timely than traditional static graphs (delay ≥ 2 hours). For example, when negative public opinion about a company spreads through the supply chain, T-GNN dynamically adds associated logistics provider nodes, and the edge weight is adjusted from the initial value of 0.2 to 0.8, shortening the warning delay from 2 hours in the traditional system to 30 seconds.

[0102] In the event chain modeling phase, event nodes are connected in series according to the "fermentation-diffusion-dissipation" stage, and the popularity trend is predicted with the help of the LSTM model. The formula is:

[0103] y t+k =LSTM(y t ,y t-1 ,...,y t-m+1 )

[0104] where y t Represents the event heat value at time t, m represents the length of the historical window, which determines the number of historical time steps used for prediction. When the threshold value θ is exceeded, a predictive warning will be triggered, where Δt represents the time interval and θ represents the threshold value dynamically adjusted by different industry characteristics. The prediction result is used to trigger a predictive warning. Trigger predictive warnings.

[0105] In this embodiment, the federated reinforcement learning warning module includes:

[0106] Local training unit: Each participant trains a Deep Q Network (DQN) based on local data, using mean squared error as the loss function for policy optimization;

[0107] Parameter aggregation unit: The central server aggregates the uploaded gradients using the FedAvg algorithm and injects Gaussian noise with variance σ = 0.1 to meet the differential privacy constraint;

[0108] Gradient Compression Unit: Retains the top 10% of key gradient parameters through Top-K sparsification technology, reducing communication overhead by more than 60%.

[0109] like Figure 3 As shown in the figure, each data participant (such as different companies, institutions, etc.) trains a deep reinforcement learning model based on the labeled multimodal data locally, using a deep Q-network algorithm. a represents the action taken (such as issuing a statement, launching an investigation, etc.), r represents the reward feedback obtained after executing the action (such as changes in public opinion heat, changes in public satisfaction, etc.), and the new state entered after executing the action is represented as s'. The experience replay buffer D is used to store the state-action-reward-next-state quadruple (s, a, r, s'). The buffer capacity can be set according to actual conditions, such as 10 5 or 10 6 etc. The loss function is defined as:

[0110] L DQN =E(s,a,r,s′)~D[(r+γ×max a′ Q(s′,a′;θ - )-Q(s,a;θ)) 2 ]

[0111] where E(s,a,r,s′)~D[·] represents the expected operation of the four-tuple (s,a,r,s′) in the experience replay buffer D; γ is the discount factor used to balance the immediate reward and long-term reward; θ is the online network parameter, θ - is the target network parameter (regularly copied and updated from the online network); s represents the current state, which contains multimodal feature information related to public opinion; a represents the action taken, such as issuing an early warning or collecting more data; r represents the reward obtained after executing action a, which is usually between 0.9 and 0.99; s' represents the next state after executing action a; Q(s, a; θ) represents the Q value determined by the online network parameter θ when executing action a in state s, that is, the estimated future cumulative reward for this action.

[0112] The central server establishes a secure communication channel with each data participant, and each participant uploads the locally trained model gradient to the central server. The central server uses differential privacy technology to add noise that follows a Laplace distribution to the gradient. The noise scale parameter is set based on the privacy protection requirements, generally between 1 and 5, to meet the differential privacy protection requirements. The FedAvg algorithm is then used to aggregate the model parameters. The specific aggregation formula is:

[0113]

[0114] where θ global represents the global model parameters after aggregation; N is the number of participants; |D i | represents the dataset size of the i-th participant; |D| represents the total size of the datasets of all participants; θ i represents the model parameters obtained by local training of the i-th participant; N((0,σ 2 I)) means the mean is 0 and the variance is σ 2 I is the Gaussian noise, where I is the identity matrix and σ is used to control the noise intensity. The aggregated gradients are sent to the gradient compression unit.

[0115] For example, when multiple financial institutions jointly train, customer transaction data is encrypted throughout the process, and the central server cannot reversely deduce the original information, thus eliminating privacy leaks from a mechanism perspective.

[0116] In this embodiment, the risk propagation quantification module quantifies the risk propagation strength of explicitly and implicitly associated entities. Explicit propagation strength is calculated as the weighted sum of the edge weights in multi-hop paths and the transaction amount ratio. Implicit propagation strength is determined by multiplying the cosine similarity between graph embedding vectors generated by Node2Vec and the industry correlation coefficient. Dynamic weight coefficients are automatically adjusted based on the stage of public opinion evolution.

[0117] Based on the multi-hop path in the knowledge graph, calculate the explicit propagation strength. Define the relationship type weight w k , such as the supply chain relationship weight is 0.8, the cooperative relationship weight is 0.6, etc. For each multi-hop path, let the path length be K, and the explicit propagation intensity is calculated by the following formula:

[0118]

[0119] where w k It is set based on the nature of the relationship between entities, reflecting the importance of different relationships in risk communication; is the ratio of the transaction amount between entity i and j under relationship k to the total transaction amount of entity i, extracted by multimodal data analysis, θ kDetermined through historical public opinion and knowledge graph feature learning, it reflects the difficulty of risk propagation under relationship k. For example, in a public opinion propagation path, there are 3 hops of relationships, the relationship type weights are 0.7, 0.6, and 0.5 respectively, the transaction amount accounts for 0.3, and the edge weights are 0.8, 0.7, and 0.6 respectively. Then the explicit propagation intensity is:

[0120] S 显式 =0.7×0.3×0.8+0.6×0.3×0.7+0.5×0.3×0.6=0.168+0.126+0.09=0.384

[0121] The graph embedding vector is generated using the Node2Vec algorithm, and the implicit propagation strength is determined by the following formula:

[0122] S 隐式 =sim(V i ,V j )×λ(t)

[0123] Where V i ,V j is the entity embedding vector generated by Node2Vec based on the knowledge graph structure and multimodal data; sim(·) is the cosine similarity function; the initial value of λ(t) is set according to industry characteristics and adjusted as public opinion evolves. For example, if the cosine similarity of the graph embedding vectors of nodes i and j is 0.7, the industry correlation coefficient is 0.8, and the current time has passed 5 hours since the event occurred, then the implicit propagation strength is:

[0124] S 隐式 =0.7×0.8×e -0.05×5 ≈0.444

[0125] The dynamic weight coefficients α(t) and β(t) are adaptively adjusted according to the stage of public opinion evolution. The formula is as follows:

[0126]

[0127] β(t)=1-α(t)

[0128] Among them, k is determined based on the historical data of industry public opinion, and t0 is the time when the public opinion breaks out; in the early stage of public opinion, β(t) dominates the exploration of potential correlations; in the later stage, α(t) dominates the accurate reflection of the actual transmission path, realizing the comprehensive quantification of the intensity of risk transmission.

[0129] Among them, k = 0.1 is the attenuation coefficient, t0 is the time of public opinion outbreak, the initial implicit weight β(t) is higher to capture potential correlations, and the later explicit weight α(t) dominates to accurately reflect the actual propagation path.

[0130] In this embodiment, the intelligent response and visualization module supports the following functions:

[0131] Propagation topology map generation: Combine entity embedding vectors and edge weights to dynamically draw an event propagation network. Node color reflects risk level (red represents the highest level), and edge transparency represents propagation intensity.

[0132] Timeline evolution diagram: key event nodes and predicted popularity trends are marked, and the dotted line represents the future development curve predicted by the LSTM model;

[0133] Tiered response strategy triggering: Automatically take corresponding measures for different levels of risk events, including freezing trading accounts, removing products from shelves, or generating public relations statement templates.

[0134] Specifically, the system generates an interactive visualization interface, including a topology map, a timeline, and a multimodal analysis report. Node color in the topology map indicates risk level (red for Level 5, orange for Level 4), and edge transparency reflects the intensity of the spread. The timeline marks key event nodes (such as the time of initial exposure and official response), and dotted lines illustrate predicted trends. An automated response engine triggers a tiered strategy based on risk level: Low-risk (Levels 1-3) generates monitoring logs every 30 minutes and pushes them to the management platform for ongoing monitoring. High-risk (Levels 4-5) automatically freezes transactions in the affected account, triggering an audit process in financial scenarios. In public relations scenarios, a pre-audit statement template is invoked and released within one hour after legal review. For example, when complaints about counterfeit goods on an e-commerce platform sparked user rights protection in multiple countries, the system linked the multilingual complaint text with comparative images of the counterfeit goods in real time, marked it as a Level 5 risk, automatically removed the product from the shelves, and notified an international legal team to intervene, improving efficiency by 80% compared to manual response.

[0135] In this embodiment, the accuracy rate of malicious public opinion identification is increased to 88%, the manual review cost is reduced by 70%, and the warning response time is shortened to 5 minutes; in Example 2, the multimodal data underreporting rate is reduced by 45%, the federated reinforcement learning communication efficiency is increased by 60%, and the automated response reduces the manual decision-making time by 80%; in Example 3, the accuracy rate of implicitly associated subject identification is increased by 35%, and the risk communication analysis time is shortened from 2 hours to 10 minutes.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning, characterized by: The system includes: a multimodal data acquisition and analysis module, a cross-modal semantic fusion module, a dynamic knowledge graph construction module, a federated reinforcement learning modeling module, a risk propagation intensity quantification module, and an intelligent response and visualization module, wherein: The multimodal data acquisition and analysis module obtains text data, visual data, and audio data information from social media, performs preliminary analysis on them, and preliminarily annotates each modal data to obtain an annotated multimodal dataset; The annotated multimodal dataset is passed to the cross-modal semantic fusion module for cross-modal risk association. The cross-modal semantic fusion module aligns the semantic features of its different modal data through a comparative learning framework and jointly determines the feature risk based on the aligned features. The dynamic knowledge graph construction module extracts entity relationships and evolution paths from the output features of the cross-modal semantic fusion module based on a temporal graph neural network to build a knowledge graph, and continuously updates the knowledge graph to reflect the development of public opinion events; The federated reinforcement learning modeling module is based on a locally trained deep reinforcement learning model and takes the event node feature data processed by the dynamic knowledge graph construction module as input. The central server aggregates model parameters through the FedAvg algorithm protected by differential privacy and secure multi-party computing, and outputs event evolution trends and risk level prediction parameters. The risk diffusion intensity quantification module is based on the event evolution path graph, combined with multi-hop path analysis and graph embedding technology. It obtains the diffusion probability and risk diffusion path intensity indicators from the entity relationship links and diffusion path node information in the knowledge graph to evaluate the potential risk diffusion capacity. The intelligent response and visualization module receives risk level prediction parameters and propagation intensity indicators, generates event evolution diagrams, time evolution response diagrams and risk control panels, and realizes the automatic triggering of risk identification, monitoring and response.

2. The multimodal public opinion risk early warning system according to claim 1, characterized in that: The multimodal data acquisition and analysis module includes: a text data processing submodule, a visual data processing submodule, and an audio data processing submodule. The text data processing submodule, the image data processing submodule, and the audio data processing submodule each have a corresponding complete data processing flow, which includes four steps: data acquisition, data preprocessing, feature extraction, and data annotation. The text data processing submodule collects text information related to public opinion, performs preprocessing operations such as denoising and word segmentation, and uses the pre-trained BERT model to extract keywords, sentiment polarity, and subject entities from text features, and annotates them based on timestamps and event tags. The image data processing submodule collects multi-source image or video data, uses the YOLOv5 model to identify sensitive targets and scene information in the image, extracts target categories, positional relationships, and image emotions from image features, and annotates them with time and event semantic information. The audio data processing submodule collects speech audio data, uses ASR technology to transcribe it into text, combines voiceprint analysis and emotion classifier to extract intonation features, speaker identity and emotional polarity in the audio features, and completes labeling based on event trigger time and content features.

3. The multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning according to claim 2 is characterized by: The cross-modal semantic fusion module maps text data, image data, and audio data into a unified semantic space based on the CLIP model, where: Set up a text encoder for text data and use the RoBERTa model to generate text vectors; Design an image encoder for image data and use ResNet-50 to output feature vector representation of the same dimension; An audio encoder is set up for audio data, and a recurrent neural network (CNN) is used to map the audio feature vector to a semantic space with the same dimension as the text and image data to generate an audio vector. Optimize its semantic alignment through contrast loss function: Wherein, the subscript i represents the current sample, j represents the comparison sample; i ′、v i ' and a i ′The aligned feature vector of the i-th text, image and audio sample; s(t i ',v i ') represents the cosine similarity between text and image features, which measures semantic consistency; s(t i ',a i ') represents the cosine similarity between text and audio features, which measures semantic consistency; τ is a temperature hyperparameter used to control the concentration of feature distribution; An exponential scaling term for text-image and text-audio feature similarity, used to calculate the comparison weights for heterogeneous samples.

4. The multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning according to claim 3 is characterized by: The process of building and updating the knowledge graph by the dynamic knowledge graph construction module is as follows: Entity node representation update unit: fuses timestamp features with entity context information to update node embedding representation; Relationship edge weight adjustment unit: uses the attention mechanism to perform temporal weighting on the edge weights in the same event chain to improve the accuracy of propagation path modeling; Event chain generation unit: Constructs an event graph based on the risk topic clustering results and updates the propagation path.

5. The multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning according to claim 4 is characterized by: In terms of node updating, the gated recurrent unit GRU and the graph convolution operation GCN are used to collaboratively update the node representation, which is specifically achieved through the following formula: in, represents the hidden state of node v at time t; represents the hidden state of node v at the last moment t-1; N(v) is the set of neighbor nodes of node v; |N(v)| is the number of neighbor nodes; W e Represents the learnable graph convolution weight matrix; W x Represents feature projection; Represents multimodal features; the update mechanism output by the cross-modal semantic fusion module is used to dynamically update the representation of nodes in the knowledge graph to reflect changes in public opinion-related information; The event chain connects event nodes in series according to the "fermentation-diffusion-dissipation" stage, and uses the LSTM model to predict the popularity trend. The formula is: y t+k =LSTM(y t ,y t-1 ,...,y t-m+1 ) where y t Represents the event heat value at time t, m represents the length of the historical window, which determines the number of historical time steps used for prediction. When a threshold value θ is exceeded, a predictive warning is triggered, where Δt represents the time interval and θ represents the threshold value dynamically adjusted based on industry characteristics. This warning provides an early signal for the system's subsequent risk response, helping the entire warning system respond to public opinion risks more promptly. It also collaborates with warnings from other subsequent modules to enhance the system's responsiveness to public opinion risks.

6. The multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning according to claim 5 is characterized by: The federated reinforcement learning warning module consists of a local training unit, a parameter aggregation unit, and a gradient compression unit. Local training unit: Each participant trains a deep Q network based on a locally stored multimodal public opinion dataset. During training, the state-action-reward-next-state quadruple (s, a, r, s') is extracted in batches from the local experience replay buffer, and the mean square error loss function L is used. DQN For strategy optimization, the formula is: L DQN =E(s,a,r,s′)~D[(r+γ×max a′ Q(s′,a′;θ - )-Q(s,a;θ)) 2 ] where E(s,a,r,s′)~D[·] represents the expected operation on the quadruple (s,a,r,s′) in the experience replay buffer D; γ is the discount factor; θ is the online network parameter, θ - is the target network parameter (periodically copied and updated from the online network); s represents the current state, which contains multimodal feature information related to public opinion; a represents the action taken; r represents the reward obtained after performing action a; s' represents the next state after performing action a; Q(s, a; θ) represents the Q value determined by the online network parameter θ when performing action a in state s, that is, the estimated future cumulative reward for this action. The gradient information generated by training is transmitted to the parameter aggregation unit; Parameter aggregation unit: The central server receives the gradient information uploaded by each participant and uses the FedAvg algorithm to calculate the weighted average for aggregation, where the weight is related to the amount of data from the participant. Gaussian noise with a variance of σ is injected into the aggregated gradient to meet the differential privacy constraint. The specific aggregation formula is: where θ global represents the global model parameters after aggregation; N is the number of participants; |D i | represents the dataset size of the i-th participant; |D| represents the total size of the datasets of all participants; θ i represents the model parameters obtained by local training of the i-th participant; N((0,σ 2 I)) means the mean is 0 and the variance is σ 2 I is the Gaussian noise, where I is the identity matrix and σ is used to control the noise intensity. The aggregated gradient is sent to the gradient compression unit; Gradient compression unit: After receiving the aggregated gradient from the parameter aggregation unit, it uses Top-K sparsification technology to retain the gradient parameters of the first K proportions, compress the communication data volume, and output the compressed sparse gradient matrix for federated model updates and iterations. Under the federated reinforcement learning framework, it achieves continuous optimization and training of the multimodal public opinion risk warning model.

7. The multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning according to claim 6 is characterized by: The risk propagation quantification module quantifies the risk propagation intensity of explicitly and implicitly associated entities in the following ways: Risk propagation quantification module: Calculates risk propagation intensity based on the data obtained by the multimodal data acquisition and analysis module and the structural information of the knowledge graph; The explicit propagation strength is calculated by the following formula: where w k The relationship between entities is set based on their nature, including supply chain relationships and cooperative relationships, to reflect the importance of different relationships in risk transmission; is the ratio of the transaction amount between entity i and j under relationship k to the total transaction amount of entity i, extracted by multimodal data analysis, θ k Determined through historical public opinion and knowledge graph feature learning, reflecting the difficulty of risk propagation under relationship k; The implicit propagation strength is determined by the following formula: S 隐式 =sim(V i ,V j )×λ(t) Where V i ,V j is the entity embedding vector generated by Node2Vec based on the knowledge graph structure and multimodal data; sim(·) is the cosine similarity function; the initial value of λ(t) is set according to the characteristics of the industry and adjusted as the public opinion evolves; the dynamic weight coefficients α(t) and β(t) are adaptively adjusted according to the stage of public opinion evolution. The formula is as follows: β(t)=1-α(t) Among them, k is determined based on the historical data of industry public opinion, and t0 is the time when the public opinion breaks out; in the early stage of public opinion, β(t) dominates the exploration of potential correlations; in the later stage, α(t) dominates the accurate reflection of the actual transmission path, realizing the comprehensive quantification of the intensity of risk transmission.

8. The multimodal public opinion risk warning system based on dynamic knowledge graph and federated reinforcement learning according to claim 7 is characterized by: The intelligent response and visualization module supports the generation of propagation topology maps, the display of timeline evolution diagrams, and the triggering of hierarchical corresponding strategies. Among them, the generation of propagation topology maps refers to the dynamic drawing of event propagation networks by combining entity embedding vectors and edge weights. The node color reflects the risk level, and the edge transparency represents the propagation intensity. The display of timeline evolution diagrams refers to the marking of key event nodes and the predicted heat trend. The dotted line represents the future development curve predicted by the LSTM model. The triggering of hierarchical response strategies refers to the automatic adoption of corresponding measures for risk events of different levels.

9. A warning method for a multimodal public opinion risk warning system based on a dynamic knowledge graph and federated reinforcement learning according to any one of claims 1 to 8, characterized in that: The method comprises: The multimodal data collection and analysis module obtains text, image, and audio data information from social media, generates a multimodal dataset through preliminary analysis and annotation, and passes it to the cross-modal semantic fusion module. The cross-modal semantic fusion module aligns the semantic features of different modalities through comparative learning, jointly judges the feature risks, and outputs the results to the dynamic knowledge graph construction module. The dynamic knowledge graph construction module, based on a time-series graph neural network, extracts entity relationships and evolution paths from input features to build a knowledge graph and continuously updates it. The output dynamic knowledge graph information (including node hidden states, event chain heat, etc.) is passed to the federated reinforcement learning modeling module. In the federated reinforcement learning modeling module, each data participant locally trains a deep reinforcement learning model based on input information, using graph features as state input, defining an action space, and receiving reward signals based on the effectiveness of responses. The central server aggregates parameters using the FedAvg algorithm, protected by differential privacy and secure multi-party computation, and outputs optimized risk level predictions and early warning strategies, which are then transmitted to the intelligent response and visualization module. Training data is also fed back to assist in optimizing the dynamic knowledge graph construction module. The risk propagation intensity quantification module combines multi-hop path analysis and graph embedding technology to evaluate the explicit and implicit strength of event propagation paths based on dynamic knowledge graphs and risk level prediction information, and quantify the risk diffusion capacity. The results are used by the intelligent response and visualization modules. The risk response and visualization display module receives risk level predictions, early warning strategies, and risk propagation intensity indicators, provides propagation topology maps, etc., triggers automated operations according to risk levels, and feeds back response results to the federated reinforcement learning modeling module to optimize strategies.

Images