Social network rumor tracing method based on multi-modal fusion and thinking chain reasoning

Abstract

The invention relates to the field of computer network application, and particularly discloses a social network rumor tracing method based on multi-modal fusion and thinking chain reasoning. Comprising the following steps: step 1, acquiring and preprocessing multi-modal information, acquiring public opinion data, and extracting a multi-modal multi-dimensional feature space; 2, performing multi-modal feature perception and knowledge fusion, learning and measuring feature isomerism among different modals, and training the sub-models layer by layer; 3, performing thinking chain reasoning, constructing a multi-level reasoning chain, extracting a feature vector, establishing a fact association according to the feature vector, synchronously constructing a fact relationship knowledge graph, and reasoning a conclusion; and 4, dynamic verification is carried out, the reasoning step text is mapped into a vector, similarity calculation is carried out, a starting threshold value is set, and when the similarity is smaller than the starting threshold value, it is judged that illusion exists in current reasoning. According to the social network rumor tracing method based on multi-modal fusion and thinking chain reasoning, the reasoning accuracy can be improved, and the credibility of a conclusion is improved.

Description

Technical Field

[0001] This invention relates to the field of computer network applications, specifically to a method for tracing the source of rumors on social networks based on multimodal fusion and thought chain reasoning. Background Technology

[0002] Rumors spread rapidly and have a wide impact on social networks. Traditional methods of rumor detection and debunking combine keyword matching and manual reporting and review. However, given the large amount of information on social networks today, these methods are quite labor-intensive.

[0003] Later, methods for detecting and debunking rumors using AI emerged, but AI-based rumor debunking has the following problems: 1) Insufficient accuracy: Rumor identification often relies on simple keyword matching, shallow neural networks or single-modal classification models. These models lack the ability to understand the deep semantics of language such as metaphor, irony, and contextual association, and cannot effectively handle the complex relationships between multimodal information, including images with misleading text and deepfake videos. Instead, they treat rumor identification as a simple binary classification task, ignoring the complex logic and factual reasoning process behind it. 2) Black box decision-making: Some high-efficiency deep learning models, such as CNN and DNN with complex structures, are themselves "black boxes" and their decision-making process is difficult to trace. 3) Rigid interaction: The design paradigm of traditional systems is "one-time question and answer", and its interaction logic is closed. The system lacks a sustainable, traceable, and queryable reasoning memory system.

[0004] The aforementioned problems result in relatively poor accuracy in identifying online rumors, and the reasoning process cannot be traced or queried, leading users to distrust the reasoning results. Summary of the Invention

[0005] The present invention aims to provide a method for tracing the source of rumors on social networks based on multimodal fusion and thought chain reasoning, so as to improve the accuracy of reasoning and the credibility of conclusions.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for tracing the source of rumors on social networks based on multimodal fusion and thought chain reasoning, comprising the following steps: Step 1: Multimodal information acquisition and preprocessing, acquire public opinion data, pre-train text and visual models respectively, and deeply extract multimodal multidimensional feature space; Step 2: Multimodal feature perception and knowledge fusion. Multimodal neural network feature learning is used to measure the feature heterogeneity between different modalities, and a layer-by-layer training algorithm is used to train the sub-model layer by layer. Step 3: Reasoning by thought chain. Construct a multi-level reasoning chain based on the big language model, extract feature vectors, establish factual associations based on feature vectors, simultaneously construct a factual relationship knowledge graph, and deduce conclusions based on the big language model. Step 4: Dynamic verification. A contrastive learning model is used to map the inference step text into vectors. The Top-K node set related to the generation step is retrieved from the rumor propagation subgraph, and similarity is calculated using the following formula 1: Formula 1 Set a threshold to determine if the similarity is less than the threshold, and then determine that the current reasoning is hallucinatory.

[0007] The beneficial effects of this plan are: 1. In this solution, when reasoning about conclusions using a large language model, in-depth analysis can be performed. Combined with the multimodal information collected in step 1, the solution not only deduce the conclusion but also simulates the cognitive process of human experts: extracting objective entities -> establishing causal / emotional connections -> constructing a factual relationship graph -> drawing conclusions. The model can handle complex causal relationships and solve the problem of "shallow logic".

[0008] Step 1, which acquires multimodal information and preprocesses the data, combined with step 2, can digest, clean, understand, and encode the chaotic, multimodal display world data into a high-quality feature vector map, thus providing a quality foundation for the completion of subsequent advanced cognitive tasks.

[0009] 2. Step 2 measures and integrates the feature heterogeneity between different modalities through attention mechanisms and layer-by-layer training algorithms. This can filter redundant, repetitive, and meaningless multimodal online public opinion big data, enhance feature recognition, and thus improve the accuracy of inference.

[0010] 3. Public opinion containing "illusions" is not considered a rumor. Traditional interaction logic is a rigid "one-time question and answer," where the model output is the final result, lacking a self-correction mechanism and prone to "illusions" or misjudgments. Step 4 of this solution uses dynamic verification to validate the conclusions inferred in step 3, checking whether the inferred content deviates from the factual constraints of the multimodal knowledge graph (MKG). Dynamic verification of the inference results in step 3 can also significantly reduce the "illusion" problem common in large models, ensuring the rigor and reliability of the conclusions.

[0011] Furthermore, it includes step 5: self-criticism, which is initiated when the similarity calculation result is less than the activation threshold. Step 5 includes judging the error type mapping and attribution. The error judgment types include factual omissions, logical contradictions, and ambiguous expressions. The normalized entropy is calculated using the following formula 2: Formula 2 Where N is the number of key tokens, V is the vocabulary space, an entropy threshold is set, and facts are judged to be missing when the value of normalized entropy is less than the entropy threshold; The final score is determined using the following formula 3: Formula 3 Set a final score threshold. If the final score is less than the final score threshold, it is considered a bit logic contradiction. If the similarity is less than the start threshold, it is considered an ambiguous expression.

[0012] The beneficial effects of this scheme are as follows: the above calculation can transform the abstract state of "knowledge scarcity" into a quantifiable indicator: when the model lacks sufficient factual knowledge to support the current generation, its predicted probability distribution tends to flatten, resulting in an increase in the calculated entropy value. By setting an "entropy threshold", when the calculation result of Formula 2 is less than the threshold, it indicates that the model exhibits abnormally "high certainty" during generation. This kind of arbitrariness in the case of insufficient evidence is precisely a typical signal of "fact-deficient" errors.

[0013] "Logical contradiction" typically manifests as a model making a very certain assertion during reasoning, but which is factually or logically flawed. This solution addresses this by setting a "final score threshold," where when M... unc If the value is less than this threshold, it indicates that the model does not exhibit sufficient abnormal certainty to trigger a contradiction alarm, thus distinguishing between "logical contradiction" and "factual absence".

[0014] Therefore, this scheme can quantify the model's "hesitation" towards the currently generated content. Through self-criticism, it can verify the conclusions obtained in step 3, thereby promptly identifying inaccurate conclusions. The final conclusions are more reliable. At the same time, the combination of formula 3 and formula 2 together achieves refined diagnosis of error types.

[0015] Furthermore, step 6 is included: dynamic correction, which dynamically assembles the error judgment type into a diagnostic report. The dynamic correction includes in-situ correction, in which the diagnostic report is attached to the context window to construct a correction prompt.

[0016] The beneficial effect of this scheme is that by combining the verification results of step 5 with the conclusion of step 3, inaccurate conclusions can be corrected, thereby further improving the reliability of the final conclusion.

[0017] Furthermore, in step 6, the error judgment type is dynamically assembled into a reflection instruction, and then the reflection instruction is output as a diagnostic report in JSON format. The diagnostic report includes the error location coordinates, error cause analysis, and suggested correction direction, and the diagnostic report is used as a signal for in-situ correction.

[0018] The beneficial effects of this solution are: the diagnostic report will not be directly output, but can be used as a control signal for in-situ correction to achieve automatic repair of the system.

[0019] Furthermore, dynamic correction also includes a backtracking operation, which presets a hyperparameter as a severe error threshold. When M... gnd A serious logical error is determined to have been found when either the serious error threshold or the error type is a logical contradiction. After completing the dynamic correction in step 6, repeat steps 4 and 5, and record the previous dynamic correction as a failure of in-situ correction when the similarity calculation result is less than the start threshold. A backtracking operation is triggered when a serious logic error is detected and K consecutive in-situ corrections fail. The backtracking operation includes the following steps: S1, Pruning: Remove the current inference branch from the search tree; S2, State Rollback: Preset high confidence threshold, when M gnd If the confidence level is greater than or equal to the high confidence threshold, the node is marked as a high confidence node, and after pruning, it is backed up to the previous high confidence node. If there is no previous high confidence node, it is backed up to the root node. S3, Temperature Perturbation Resampling: Temporarily increase the decoding temperature parameter to explore inference paths that have not been pruned.

[0020] Furthermore, step S3 outputs the path that successfully traces back to the root node and has the highest cumulative confidence level as the final evidence chain for tracing the source of the rumor.

[0021] The beneficial effects of this scheme are: by backtracking, the model can be guided to explore different reasoning paths, escape the trap of local optima, and thus obtain more accurate conclusions.

[0022] Furthermore, it also includes step 7: visual tracing, which forms a tree diagram of the reasoning process in steps 3 and 4, sets up a display window, and links the display window with the evidence chain for tracing the source of the rumor.

[0023] The beneficial effects of this solution are as follows: It establishes a link between the user and the reasoning process, concretizing the abstract process of public opinion reasoning within a large language model. It clearly demonstrates the process of information gathering, integrating, and reasoning from different sources, facilitating the understanding of complex public opinion reasoning logic. It presents the multi-step nature of the reasoning process, with each stage of the technical route displayed in a tree-like diagram for source tracing and localization. Presenting complex information intuitively to users breaks the problem of black-box conclusions in traditional AI-based rumor debunking, while improving information delivery efficiency. Furthermore, in traditional rumor reasoning, users cannot know why they were / were not judged as spreading rumors, resulting in a poor interactive experience and difficulty in building trust. This solution establishes a display window to synchronously display the reasoning process, enhancing the user experience.

[0024] Furthermore, it also includes step 8: human-machine feedback, receiving human feedback and feeding the feedback results back to step 2.

[0025] The beneficial effect of this scheme is that it enables periodic iterative optimization of the model.

[0026] Furthermore, the feature vector in step 3 includes one or more of the following: entity, event, time, and location.

[0027] The beneficial effect of this approach is that extracting as many different feature vectors as possible is more conducive to inferring more accurate conclusions.

[0028] Furthermore, in step 1, public opinion data is obtained through web scraping technology, and dynamic interactive information is retained.

[0029] The beneficial effects of this solution are as follows: Traditional methods often rely on simple keyword matching or shallow neural networks, treating the task as a single modality (plain text), leading to severe heterogeneity of information sources and significant noise interference. This solution, when acquiring multi-source public opinion data such as text, images, and videos, captures content while retaining dynamic interactive information, enabling the identification of deep connections between modalities and effectively solving the problem of "feature heterogeneity and semantic disconnection." Attached Figure Description

[0030] Figure 1 This is a flowchart of steps 4 to 6 in an embodiment of the present invention; Figure 2 This is a flowchart of step 7 in an embodiment of the present invention. Detailed Implementation

[0031] The following detailed description illustrates the specific implementation method: Example The embodiments disclose a method for tracing the source of rumors on social networks based on multimodal fusion and thought chain reasoning, combining... Figure 1 and picture Figure 2 This includes the following steps: Step 1: Multimodal information acquisition and preprocessing. Obtain public opinion data through web crawling technology. The acquisition objects include, but are not limited to, web pages, apps and other channels. The public opinion data includes, but is not limited to, text, images and videos, and retains dynamic interactive information. Pre-train text models and visual models respectively, and deeply extract multimodal multidimensional feature spaces. Step 2: Multimodal feature perception and knowledge fusion. Multimodal neural network feature learning is used to measure the feature heterogeneity between different modalities, and a layer-by-layer training algorithm is used to train the sub-model layer by layer. Step 3: Reasoning by thought chain. Construct a multi-level reasoning chain based on the large language model, extract feature vectors, which include one or more of entities, events, time, and location; establish factual associations based on feature vectors, simultaneously construct a factual relationship knowledge graph, and deduce conclusions based on the large language model. Step 4: Dynamic verification. A contrastive learning model is used to map the inference step text into vectors. The Top-K node set related to the generation step is retrieved from the rumor propagation subgraph, and similarity is calculated using the following formula 1: Formula 1 Set a threshold to determine if the current reasoning is hallucinatory when the similarity is less than the threshold. Step 5: Self-criticism, initiated when the similarity calculation result is less than the activation threshold. Step 5 includes determining the error type mapping and attribution. The error judgment types include factual omissions, logical contradictions, and ambiguous statements. The normalized entropy is calculated using the following formula 2: Formula 2 Where N is the number of key tokens, V is the vocabulary space, an entropy threshold is set, and facts are judged to be missing when the value of normalized entropy is less than the entropy threshold; The final score is determined using the following formula 3: Formula 3 Set a final score threshold; if the final score is less than the final score threshold, it is considered a bit logic contradiction; and if the similarity is less than the start threshold, it is considered an ambiguous expression. The start threshold, entropy threshold, and final score threshold are constants set by the administrator, and in this embodiment, they are set to 0.6, 0.2, and 0.7, respectively. Step 6: Dynamic correction. Dynamic correction includes in-situ correction and backtracking operations. Metacognitive prompting technology is used to dynamically assemble the error judgment type, reasoning step text, and related evidence into reflection instructions. The reflection instructions are then input into the LLM and finally output a diagnostic report in JSON format. The diagnostic report includes whether the error is repairable, the error location coordinates, the error cause analysis, and the suggested correction direction. The diagnostic report serves as a signal for in-situ correction. When the error is diagnosed as repairable, in-situ correction is initiated. During in-situ correction, the diagnostic report is attached to the context window to construct correction prompts. When performing a backtracking operation, a preset critical error threshold is a hyperparameter. When M gnd A serious logical error is determined to have been found when either the serious error threshold or the error type is a logical contradiction. After completing the dynamic correction in step 6, repeat steps 4 and 5, and record the previous dynamic correction as a failure of in-situ correction when the similarity calculation result is less than the start threshold. A backtracking operation is triggered when a serious logical error is detected and K consecutive in-situ corrections fail. In actual implementation, K is an integer greater than zero. The backtracking operation includes the following steps: S1, Pruning: Mark the current inference branch as a "dead end" and prune it from the search tree; S2, State Rollback: Preset high confidence threshold, when M gnd If the confidence level is greater than or equal to the high confidence threshold, the node is marked as a high confidence node, and after pruning, it is backed up to the previous high confidence node. If there is no previous high confidence node, it is backed up to the root node. S3, Temperature Perturbation Resampling: Temporarily increase the decoding temperature parameter, explore unpruned inference paths, maintain a candidate path set, and output the path that successfully traces back to the root node and has the highest cumulative confidence as the final rumor tracing evidence chain; Step 7: Visualize the source tracing. Form a tree diagram of the reasoning process in Steps 3 and 4, set up a display window, and link the display window with the evidence chain for tracing the source of the rumor. Step 8: Human-machine feedback, receive human feedback and send the feedback results back to Step 2.

[0032] The dynamic verification mechanism used in step 4 of this embodiment is based on the construction of a fact consistency constraint loop: 1. By using a contrastive learning model, the inference text of each step of the large language model is encoded into vectors, and the most relevant Top-K evidence nodes are retrieved in the "rumor propagation subgraph" constructed based on real data. This process essentially anchors and aligns the open-ended language inference within the model with the closed-ended factual knowledge graph outside.

[0033] 2. Similarity calculation quantifies "illusion risk": Formula 1 calculates the maximum semantic similarity between the current reasoning step and the most relevant known evidence. M gnd The higher the value, the stronger the factual basis supporting the reasoning step; conversely, when M... gnd If the threshold is lower than the set threshold, it strongly suggests that the model may have deviated from factual constraints and produced "illusions" or conjectures based on internal parameters rather than external evidence.

[0034] 3. Real-time monitoring and correction: This mechanism does not only verify the final conclusion, but also performs real-time factual compliance checks at every step of the reasoning chain. This is equivalent to implanting a continuously running "fact checker" in the reasoning process, which can immediately identify and interrupt reasoning paths that deviate from the facts, thereby ensuring the accuracy of the final conclusion from the process and solving the core pain points of uncontrollable and easily "illusory" content generated by large models.

[0035] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for tracing the origins of rumors on social networks based on multimodal fusion and thought chain reasoning, characterized by: Includes the following steps: Step 1: Multimodal information acquisition and preprocessing, acquire public opinion data, pre-train text and visual models respectively, and deeply extract multimodal multidimensional feature space; Step 2: Multimodal feature perception and knowledge fusion. Multimodal neural network feature learning is used to measure the feature heterogeneity between different modalities, and a layer-by-layer training algorithm is used to train the sub-model layer by layer. Step 3: Reasoning by thought chain. Construct a multi-level reasoning chain based on the big language model, extract feature vectors, establish factual associations based on feature vectors, simultaneously construct a factual relationship knowledge graph, and deduce conclusions based on the big language model. Step 4: Dynamic verification. A contrastive learning model is used to map the inference step text into vectors. The Top-K node set related to the generation step is retrieved from the rumor propagation subgraph, and similarity is calculated using the following formula 1: Official 1 in, Represents nodes in a subgraph ; The vector representing the inference steps text is obtained by mapping it through a contrastive learning model. This represents a complete sub-graph depicting the spread of a rumor. Set a threshold to determine if the similarity is less than the threshold, and then determine that the current reasoning is hallucinatory.

2. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 1, characterized in that: This includes step 5: self-criticism, which is initiated when the similarity calculation result is less than the activation threshold. Step 5 includes judging the error type mapping and attribution. The error judgment types include factual omissions, logical contradictions, and ambiguous expressions. The normalized entropy is calculated using the following formula 2: Official 2 Where N is the number of key tokens, V is the vocabulary space, an entropy threshold is set, and facts are judged to be missing when the value of normalized entropy is less than the entropy threshold; The final score is determined using the following formula 3: Official 3 Set a final score threshold. If the final score is less than the final score threshold, it is considered a bit logic contradiction. If the similarity is less than the start threshold, it is considered an ambiguous expression.

3. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 2, characterized in that: Step 6: Dynamic correction, which dynamically assembles the error judgment type into a diagnostic report. The dynamic correction includes in-situ correction, in which the diagnostic report is attached to the context window to construct a correction prompt.

4. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 3, characterized in that: Step 6 first dynamically assembles the error judgment type into a reflection instruction, and then outputs the reflection instruction as a diagnostic report in JSON format. The diagnostic report includes the error location coordinates, error cause analysis, and suggested correction direction, and serves as a signal for in-situ correction.

5. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 3, characterized in that: Dynamic correction also includes a backtracking operation, which presets a hyperparameter as a severe error threshold. When M gnd A serious logical error is determined to have been found when either the serious error threshold or the error type is a logical contradiction. After completing the dynamic correction in step 6, repeat steps 4 and 5, and record the previous dynamic correction as a failure of in-situ correction when the similarity calculation result is less than the start threshold. A backtracking operation is triggered when a serious logic error is detected and K consecutive in-situ corrections fail. Includes the following steps: S1, Pruning: Remove the current inference branch from the search tree; S2, State Rollback: Preset high confidence threshold, when M gnd If the confidence level is greater than or equal to the high confidence threshold, the node is marked as a high confidence node, and after pruning, it is backed up to the previous high confidence node. If there is no previous high confidence node, it is backed up to the root node. S3, Temperature Perturbation Resampling: Temporarily increase the decoding temperature parameter to explore inference paths that have not been pruned.

6. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 5, characterized in that: Step S3 outputs the path that successfully traces back to the root node and has the highest cumulative confidence level as the final chain of evidence for tracing the source of the rumor.

7. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 6, characterized in that: It also includes step 7: visual tracing, which forms a tree diagram of the reasoning process in steps 3 and 4, sets up a display window, and links the display window to the evidence chain for tracing the source of the rumor.

8. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 7, characterized in that: It also includes step 8: human-machine feedback, receiving human feedback and feeding the feedback results back to step 2.

9. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 1, characterized in that: The feature vector in step 3 includes one or more of the following: entity, event, time, and location.

10. The method for tracing the origin of social network rumors based on multimodal fusion and thought chain reasoning as described in claim 1, characterized in that: Step 1: Obtain public opinion data through web scraping technology and retain dynamic interactive information.

Images