Multi-modal social network public opinion hidden danger checking method and system

Abstract

The invention discloses a multi-modal social network public opinion hidden danger troubleshooting method and system, and the method comprises the steps: collecting the multi-modal data of a social platform in real time through a distributed edge node, and the multi-modal data comprise text, image and audio data; dividing a plurality of social network communities based on user interaction relationships, wherein the user interaction relationships include but are not limited to topic circles, friend relationships, comment interaction and forwarding likes; community propagation dynamic features, group topological features and content features are extracted from the multiple social network communities one by one based on the multi-modal data; screening out a plurality of suspected hidden danger communities from the plurality of social network communities based on the community propagation dynamic characteristics, the group topological characteristics, the content characteristics and a preset screening strategy; and after carrying out hidden danger detection and hidden danger classification on the plurality of suspected hidden danger communities based on the multi-modal data, outputting a plurality of hidden danger classifications corresponding to the plurality of suspected hidden danger communities. According to the method and the system, the comprehensiveness and the accuracy of public opinion hidden danger recognition are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of social network public opinion monitoring, and in particular to a method and system for troubleshooting multimodal social network public opinion hidden dangers. Background Art

[0002] Current social network public opinion monitoring technologies primarily utilize a single-modal data processing model, parsing independent data types such as text, images, audio, or video. Analysis is performed by extracting textual information, but they lack a collaborative processing mechanism for multimodal data. This technical architecture limits public opinion analysis to a single data format and fails to effectively capture the correlations between textual semantics, visual features, and auditory information. Regarding dynamic propagation analysis, existing methods fail to adequately model the topological evolution of social networks and lack quantitative analysis of key characteristics such as the dynamic changes in public opinion propagation paths and the diffusion of influence, resulting in insufficient predictive model accuracy. Operational processes primarily rely on a serial processing model with manual intervention, resulting in gaps in the automated analysis chain and limited cross-platform data synchronization and real-time computing capabilities, making them unable to adapt to the rapidly evolving demands of public opinion trends. Specifically, in multimedia content recognition, existing systems lack robust detection algorithms for unstructured data such as sensitive image features and prohibited voiceprint features. Furthermore, they lack models for correlating user interactions with content features, leading to a significant risk of missed detections in multimodal public opinion scenarios. Summary of the Invention

[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a method and system for troubleshooting public opinion risks in a multimodal social network.

[0004] The present invention proposes a method for troubleshooting multimodal social network public opinion risks, comprising the following steps:

[0005] S1. Collect multimodal data from social platforms in real time through distributed edge nodes, where the multimodal data includes text, images, and audio data.

[0006] S2. Divide multiple social network communities based on user interaction relationships, including but not limited to topic circles, friend relationships, comment interactions, and forwarding and liking.

[0007] S3. Extract community communication dynamics, group topology, and content features from multiple social network communities based on multimodal data.

[0008] S4. Based on the community transmission dynamics characteristics, group topology characteristics, content characteristics and preset screening strategies, multiple suspected risk communities are screened out from multiple social network communities;

[0009] S5. After performing hidden danger detection and hidden danger classification on multiple suspected hidden danger communities based on multimodal data, multiple hidden danger classifications corresponding to the multiple suspected hidden danger communities are output, where the hidden danger classifications include high risk, warning, and concern.

[0010] Preferably, the community communication dynamics characteristics specifically include the time derivative of the forwarding rate and the network modularity; the group topology characteristics specifically include the user clustering Gini coefficient and the sentiment polarization index; the content characteristics include the density of sensitive words and the probability of visual prohibition.

[0011] Preferably, the preset screening strategy specifically includes:

[0012] Compare the community transmission dynamics, group topology, and content characteristics with the corresponding preset screening threshold ranges one by one;

[0013] When any of the community communication dynamics characteristics, group topology characteristics and content characteristics does not meet the preset screening threshold range, the corresponding social network community will be regarded as a suspected risk community.

[0014] Preferably, the hidden danger detection in step S5 specifically includes:

[0015] Based on multimodal data, cross-modal verification is performed on user content in multiple communities suspected of potential risks. The semantic consistency between text and image or audio data in the community is analyzed through a pre-trained two-stream cross-validation network, and cross-modal contradictory content is marked as high-risk nodes to obtain cross-modal verification results.

[0016] Preferably, marking cross-modal contradictory content as a high-risk node requires satisfying any of the following conditions:

[0017] (a) The prohibited probability of the image / audio is greater than the threshold P, and the semantics of the text description is inconsistent with the prohibited content, that is, the semantic similarity is less than the threshold Q;

[0018] (b) The independent prohibited probabilities of text, image, and audio are all greater than the threshold R, and the cross-modal attention weight is greater than the threshold S;

[0019] (c) The difference in the unimodal prohibited probability is greater than the threshold T, and the cross-modal semantic similarity is less than the threshold U.

[0020] Preferably, the hidden danger classification in step S5 specifically includes:

[0021] The community transmission dynamics characteristics, group topology characteristics, and cross-modal verification results are input into the pre-trained LightGBM hierarchical classifier to output hidden danger classification, which includes high risk, warning, and concern.

[0022] Preferably, the method further includes: after step S5:

[0023] S6: Conduct propagation simulations for communities classified as high-risk, encode propagation paths using the Time-LLM time series model, and simulate propagation topology and impact range using graph transformation networks and neural differential equations.

[0024] S7. Generate an intervention strategy based on the deduction results, wherein the intervention strategy includes content deletion, user guidance, or official rumor refutation.

[0025] Preferably, the propagation deduction in step S6 supports minute-level trajectory prediction, specifically including: generating a heat map of the propagation of the primary event for the unintervention community; and generating a waveform map of the duration of the prevention and control event for the intervention community.

[0026] The present invention proposes a multimodal social network public opinion risk investigation system, comprising:

[0027] A data collection module, which is used to collect multimodal data of the social platform in real time through distributed edge nodes. The multimodal data includes text, images, audio and video data;

[0028] A community segmentation module is used to segment multiple social network communities based on user interaction relationships, including but not limited to topic circles, friend relationships, comment interactions, and forwarding and liking;

[0029] Feature extraction module, used to extract community communication dynamics characteristics, group topology characteristics and content characteristics of multiple social network communities one by one based on multimodal data;

[0030] A community screening module is used to screen out multiple suspected risk communities from multiple social network communities based on community communication dynamics, group topology, content characteristics, and preset screening strategies;

[0031] The hidden danger detection and classification module is used to detect and classify hidden dangers in multiple communities suspected of hidden dangers based on multimodal data, and output multiple hidden danger classifications corresponding to the multiple suspected hidden danger communities. The hidden danger classifications include high risk, warning, and concern.

[0032] Preferably, it also includes:

[0033] The propagation deduction module is used to deduce the spread of hidden dangers in communities classified as high-risk. The propagation path is encoded through the Time-LLM time series model. The propagation topology and impact range are simulated based on graph transformation networks and neural differential equations.

[0034] The intervention module is used to generate an intervention strategy based on the deduction results, and the intervention strategy includes content deletion, user guidance or official rumor refutation.

[0035] In the present invention, the proposed multimodal social network public opinion risk investigation method and system collects multimodal data through distributed edge nodes; divides communities based on user interaction relationships, extracts propagation dynamics, group topology and content features, and screens suspected risk communities; uses a two-stream cross-validation network to achieve cross-modal contradiction detection, and combines the LightGBM classifier to output three-level labels of high risk, warning, and attention; further simulates the propagation path through Time-LLM and Graph Transformer to generate intervention strategies; finally, optimizes the system parameters through the closed-loop effect evaluation intelligent agent. This significantly improves the comprehensiveness and accuracy of public opinion risk identification and solves the limitations of single-modal data processing in existing technologies. It also makes up for the shortcomings of existing technologies in propagation dynamics modeling, making the prediction of public opinion risks more scientific and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the workflow of a multimodal social network public opinion risk investigation method proposed by the present invention;

[0037] Figure 2 This is a structural diagram of an implementation method of a multimodal social network public opinion risk investigation method proposed by the present invention;

[0038] Figure 3 This is a schematic diagram of the system architecture of a multimodal social network public opinion risk investigation system proposed by the present invention. DETAILED DESCRIPTION

[0039] Reference Figure 1-3 The present invention proposes a method for troubleshooting potential risks of public opinion in a multimodal social network, comprising the following steps:

[0040] S1. Collect multimodal data of social platforms in real time through distributed edge nodes. Multimodal data includes text, images, and audio data.

[0041] S2. Divide multiple social network communities based on user interaction relationships, where user interaction relationships include but are not limited to topic circles, friend relationships, comment interactions, and forwarding and liking.

[0042] In this embodiment, the division methods corresponding to dividing multiple social network communities based on user interaction relationships include but are not limited to partitioning algorithms such as modularity optimization algorithm, edge betweenness algorithm, spectral clustering algorithm, and graph convolutional network.

[0043] S3. Extract community communication dynamics characteristics, group topology characteristics and content characteristics of multiple social network communities one by one based on multimodal data.

[0044] Specifically, methods for extracting community communication dynamics features from multiple social network communities one by one include but are not limited to extracting text, image, and audio data using domain-adapted vector libraries, feature-level fusion, and other methods.

[0045] In this embodiment, the community communication dynamics characteristics specifically include the time derivative of the forwarding rate and the network modularity; the group topology characteristics specifically include the user clustering Gini coefficient and the sentiment polarization index; and the content characteristics include the density of sensitive words and the probability of visual prohibition.

[0046] S4. Based on the community communication dynamics characteristics, group topology characteristics, content characteristics and preset screening strategies, multiple suspected hidden danger communities are screened out from multiple social network communities.

[0047] In this embodiment, the preset screening strategy specifically includes:

[0048] Compare the community transmission dynamics, group topology, and content characteristics with the corresponding preset screening threshold ranges one by one;

[0049] When any of the community communication dynamics characteristics, group topology characteristics and content characteristics does not meet the preset screening threshold range, the corresponding social network community will be regarded as a suspected risk community.

[0050] S5. After performing hidden danger detection and classification on multiple communities suspected of hidden dangers based on multimodal data, multiple hidden danger classifications corresponding to the multiple communities suspected of hidden dangers are output. The hidden danger classifications include high risk, warning, and concern.

[0051] In this embodiment, the hidden danger detection in step S5 specifically includes:

[0052] Based on multimodal data, cross-modal verification is performed on user content in multiple communities suspected of potential risks. The semantic consistency between text and image or audio data in the community is analyzed through a pre-trained two-stream cross-validation network, and cross-modal contradictory content is marked as high-risk nodes to obtain cross-modal verification results.

[0053] Specifically, marking cross-modal contradictory content as a high-risk node requires any of the following conditions to be met:

[0054] (a) The prohibited probability of the image / audio is greater than the threshold P, and the semantics of the text description is inconsistent with the prohibited content, that is, the semantic similarity is less than the threshold Q;

[0055] (b) The independent prohibited probabilities of text, image, and audio are all greater than the threshold R, and the cross-modal attention weight is greater than the threshold S;

[0056] (c) The difference in the unimodal prohibited probability is greater than the threshold T, and the cross-modal semantic similarity is less than the threshold U.

[0057] It should be noted that multiple communities suspected of potential risks are the focus of the detection agent's investigation. High-dimensional risk detection is performed on these communities, examining user content risk characteristics such as sensitive word density, visual violation probability, and voiceprint violation, to deeply detect risky individuals or objects. Simultaneously, a two-stream cross-validation network is used to analyze the semantic consistency between text descriptions and image or audio content, enabling cross-modal risk verification. Users with content risks are marked as risky nodes within the community.

[0058] Specifically, the training process of the two-stream cross-validation network is to extract features in parallel through bimodal streams and exchange to generate adversarial supervision signals to iteratively optimize the cross-modal consistency loss.

[0059] In this embodiment, the hidden danger classification in step S5 specifically includes:

[0060] The community transmission dynamics characteristics, group topology characteristics, and cross-modal verification results are input into the pre-trained LightGBM hierarchical classifier to output hidden danger classification, which includes high risk, warning, and concern.

[0061] In this embodiment, the following steps are further included after step S5:

[0062] S6: Conduct propagation simulations for communities classified as high-risk, encode propagation paths using the Time-LLM time series model, and simulate propagation topology and impact range using graph transformation networks and neural differential equations.

[0063] S7. Generate intervention strategies based on the deduction results, including content deletion, user guidance, or official rumor refutation.

[0064] Specifically, the propagation deduction in step S6 supports minute-level trajectory prediction, including: generating a heat map of the propagation of the primary event for the unintervention community; and generating a waveform diagram of the duration of the prevention and control event for the intervention community.

[0065] Specifically, we extract surface features, contextual features, and community features from multimodal data to construct a high-dimensional feature vector dataset of hidden dangers. Surface features include, but are not limited to, keywords and visual elements; contextual features include, but are not limited to, spatiotemporal tags and device fingerprints; and community features include, but are not limited to, user profiles and group behavior patterns. We also introduce the Time-LLM time series model, encoding propagation path data into a learnable token sequence. Using a spatiotemporal attention mechanism, we analyze the spatiotemporal dependencies of cross-platform propagation.

[0066] Specifically, the core idea of ​​the Graph Transformer is to apply the Transformer architecture to graph-structured data, and to achieve inter-node relationship processing and long-range dependency capture by integrating the Graph Neural Network (GNN) with the self-attention mechanism. Graph Transformer is used to build a dynamic evolution engine, combined with a neural differential equation solver to achieve continuous-time deduction of hidden danger states, supporting minute-level propagation trajectory prediction and intervention strategy simulation. Supporting minute-level propagation trajectory prediction specifically includes: for hidden danger communities that have not been intervened, simulating the propagation path topology, impact range heat map, and duration waveform after the labeled users in the community send hidden danger information, and generating primary hidden danger propagation events. For high-risk communities after the implementation of the intervention strategy, the propagation path, impact range, and duration of the hidden danger information are simulated again to generate hidden danger prevention and control propagation events.

[0067] Specifically, intervention strategies include:

[0068] a. Content management: Deleting or restricting the dissemination of illegal content.

[0069] b. User guidance: Warning or educating users who post sensitive information.

[0070] c. Official rumor refutation: Release authoritative information to guide the public to view the incident correctly.

[0071] d. Social network regulation: Adjust the dissemination mechanism of social networks to reduce the speed of dissemination of potential danger information.

[0072] In this embodiment, the method further includes: generating a hidden danger investigation work report by combining the original hidden danger propagation event without intervention and the hidden danger propagation event with intervention.

[0073] This embodiment also includes a closed-loop feedback core process for the intelligent agent group, which is responsible for real-time monitoring, data analysis, and effect evaluation of the entire process of hidden danger investigation, deduction, and decision-making. The generated strategies can be evaluated, and the simulated review team will review them and provide opinions. If they fail, they will be sent back to the process and re-deduced. Based on public opinion feedback after real hidden dangers occur, each link of the system is quantitatively evaluated, providing data support for system optimization and improvement, ensuring that the system can be continuously iterated and optimized, forming a complete closed-loop feedback mechanism, and ensuring the efficient and accurate operation of the system.

[0074] Reference Figure 1-3 The present invention proposes a multimodal social network public opinion risk investigation system, comprising:

[0075] The data collection module is used to collect multimodal data from social platforms in real time through distributed edge nodes. Multimodal data includes text, images, audio, and video data;

[0076] a community division module configured to divide a plurality of social network communities based on user interaction relationships, the user interaction relationships including but not limited to topic circles, friend relationships, comment interactions, and forwarding likes;

[0077] a feature extraction module configured to extract community propagation dynamics features, group topology features, and content features from the plurality of social network communities based on multi-modal data;

[0078] a community screening module configured to screen a plurality of suspected hidden danger communities from the plurality of social network communities based on the community propagation dynamics features, the group topology features, the content features, and a preset screening strategy;

[0079] a hidden danger detection and classification module configured to output a plurality of hidden danger classifications corresponding to the plurality of suspected hidden danger communities after hidden danger detection and hidden danger classification are performed on the plurality of suspected hidden danger communities based on multi-modal data, the hidden danger classifications including high risk, early warning, and attention.

[0080] In the embodiment, the method further includes:

[0081] a propagation deduction module configured to perform propagation deduction on a community corresponding to the hidden danger classification of high risk, encode a propagation path through a Time-LLM time series model, and simulate propagation topology and an influence range based on a graph transformation network and a neural differential equation;

[0082] an intervention module configured to generate an intervention strategy according to a deduction result, the intervention strategy including content deletion, user guidance, or official refutation.

[0083] The above description is only a preferred embodiment of the present application, and the protection scope of the present application is not limited thereto. Any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the present application within the technical scope disclosed by the present application, and such replacements or changes should be covered within the protection scope of the present application.

Claims

1. A method for troubleshooting hidden dangers of public opinion in multimodal social networks, characterized by: The following steps are involved: S1. Collect multimodal data from social platforms in real time through distributed edge nodes, where the multimodal data includes text, images, and audio data. S2. Divide multiple social network communities based on user interaction relationships, including but not limited to topic circles, friend relationships, comment interactions, and forwarding and liking. S3. Extract community communication dynamics, group topology, and content features from multiple social network communities based on multimodal data. S4. Based on the community transmission dynamics characteristics, group topology characteristics, content characteristics and preset screening strategies, multiple suspected risk communities are screened out from multiple social network communities; S5. After performing hidden danger detection and hidden danger classification on multiple suspected hidden danger communities based on multimodal data, multiple hidden danger classifications corresponding to the multiple suspected hidden danger communities are output, where the hidden danger classifications include high risk, warning, and concern.

2. The method for troubleshooting multimodal social network public opinion risks according to claim 1, characterized in that: The community communication dynamics characteristics specifically include the time derivative of the forwarding rate and the network modularity; the group topology characteristics specifically include the user clustering Gini coefficient and the sentiment polarization index; the content characteristics specifically include the sensitive word density and the visual violation probability.

3. The method for troubleshooting multimodal social network public opinion risks according to claim 2, characterized in that: The preset screening strategy specifically includes: Compare the community transmission dynamics, group topology, and content characteristics with the corresponding preset screening threshold ranges one by one; When any of the community communication dynamics characteristics, group topology characteristics and content characteristics does not meet the preset screening threshold range, the corresponding social network community will be regarded as a suspected risk community.

4. The method for troubleshooting multimodal social network public opinion risks according to claim 1, characterized in that: The hidden danger detection in step S5 specifically includes: Based on multimodal data, cross-modal verification is performed on user content in multiple communities suspected of potential risks. The semantic consistency between text and image or audio data in the community is analyzed through a pre-trained two-stream cross-validation network, and cross-modal contradictory content is marked as high-risk nodes to obtain cross-modal verification results.

5. The method for troubleshooting multimodal social network public opinion risks according to claim 4, characterized in that: The marking of cross-modal contradictory content as a high-risk node must meet any of the following conditions: (a) The prohibited probability of the image / audio is greater than the threshold P, and the semantics of the text description is inconsistent with the prohibited content, that is, the semantic similarity is less than the threshold Q; (b) The independent prohibited probabilities of text, image, and audio are all greater than the threshold R, and the cross-modal attention weight is greater than the threshold S; (c) The difference in the unimodal prohibited probability is greater than the threshold T, and the cross-modal semantic similarity is less than the threshold U.

6. The method for troubleshooting multimodal social network public opinion risks according to claim 4, characterized in that: The hidden danger classification in step S5 specifically includes: The community transmission dynamics characteristics, group topology characteristics, and cross-modal verification results are input into the pre-trained LightGBM hierarchical classifier to output hidden danger classification, which includes high risk, warning, and concern.

7. The method for troubleshooting multimodal social network public opinion risks according to claim 1, characterized in that: Also includes: After step S5, the method further includes: S6: Conduct propagation simulations for communities classified as high-risk, encode propagation paths using the Time-LLM time series model, and simulate propagation topology and impact range using graph transformation networks and neural differential equations. S7. Generate an intervention strategy based on the deduction results, wherein the intervention strategy includes content deletion, user guidance, or official rumor refutation.

8. The method for troubleshooting multimodal social network public opinion risks according to claim 7, characterized in that: The propagation deduction in step S6 supports minute-level trajectory prediction, specifically including: generating a heat map of the primary event propagation for the unintervention community; and generating a waveform map of the duration of the prevention and control event for the intervention community.

9. A multimodal social network public opinion risk investigation system, characterized by: include: A data collection module, which is used to collect multimodal data of the social platform in real time through distributed edge nodes. The multimodal data includes text, images, audio and video data; A community segmentation module is used to segment multiple social network communities based on user interaction relationships, including but not limited to topic circles, friend relationships, comment interactions, and forwarding and liking; Feature extraction module, used to extract community communication dynamics characteristics, group topology characteristics and content characteristics of multiple social network communities one by one based on multimodal data; The community screening module is used to screen out multiple suspected risk communities from multiple social network communities based on community communication dynamics, group topology, content characteristics, and preset screening strategies; The hidden danger detection and classification module is used to detect and classify hidden dangers in multiple communities suspected of hidden dangers based on multimodal data, and output multiple hidden danger classifications corresponding to the multiple suspected hidden danger communities. The hidden danger classifications include high risk, warning, and concern.

10. The multimodal social network public opinion risk investigation system according to claim 9, characterized in that: Also includes: The propagation deduction module is used to perform propagation deduction on communities classified as high-risk by using the Time-LLM time series model to encode the propagation path; Simulate the propagation topology and influence range based on graph transformation networks and neural differential equations; The intervention module is used to generate an intervention strategy based on the deduction results, wherein the intervention strategy includes content deletion, user guidance or official rumor refutation.

Images