Counterfeit information detection method combining bimodal understanding and large language model
By combining bimodal understanding and a fake information detection method of large language models, the problems of insufficient structural evidence and insufficient explanation of modal relationships in false news detection are solved, and more accurate judgments on the authenticity of news and logical reasoning are achieved. Reduced labor costs.
Patent Information
- Application Number
- CN202410996900.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art has problems in the detection of false news, lack of structural evidence for retrieval, hallucination of model reasoning, and insufficient interpretation of modal relationships between images and texts, and building knowledge graphs requires a lot of labor costs and time.
Using a forged information detection method combining bimodal understanding and large language models, the modal correlation between image and text is explained through multi-grained thinking chains, search titles and phrases are generated to search evidence, extract entities and relationships and represent them as knowledge graph triplets , use large language models to make inference judgments.
Improve the interpretation of correlation between modals, enhance the accuracy of the judgment of true and false news, avoid model reasoning illusions, provide structured evidence representation, make full use of the advantages of knowledge graphs and large language models, and improve reasoning logic.
Smart Images

Figure CN119003767A_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a forged information detection method combining bimodal understanding with a large language model. [Background Technology]
[0002] As social media platforms are increasingly integrated into people's lives, they have become the main source of information for the public. Therefore, false news can be spread quickly, and the content of false news can easily confuse and mislead people, which in turn affects people's judgment and decision.
[0003] Multimodal large language models, such as GPT4 and llava, have powerful reasoning capabilities, zero-sample learning capabilities, and multimodal reasoning and judgment capabilities. They are widely used in forged information detection, but they still have the following problems:
[0004] 1. Although many studies on fake news detection use search engines to retrieve external evidence and use large language models (LLMs) to combine evidence with image-text modal reasoning and judgment, the retrieved evidence is mostly in the form of text and lacks structure, and the reasoning of large language models (LLMs) will produce hallucinations.
[0005] 2. Some studies on fake news detection based on large language models (LLMs) have failed to provide a reasonable explanation for the modal relationship between images and texts. Currently, many studies use contrastive learning methods, such as aligning image and text modalities, but the limitations of this method are long model training time, low efficiency, and lack of explainability of the relationship between modalities.
[0006] 3. The key to building a knowledge graph KG is to effectively extract entities and relationships, but identifying entities and the relationships between them requires knowledge of various fields of society, which requires a lot of manpower costs and is time-consuming. [Summary of the invention]
[0007] The present invention overcomes the deficiencies of the prior art and provides a method for detecting forged information that combines bimodal understanding with a large language model.
[0008] To achieve the above object, the present invention adopts the following technical solutions:
[0009] A method for detecting forged information combining bimodal understanding and a large language model, characterized in that:
[0010] S1. Multimodal understanding, including noun interpretation perspective, sentence perspective and multimodal perspective, uses multi-granularity thinking chains to explain the correlation between image modality and text modality through a large language model, and determines whether the image is useful for the text;
[0011] S2. Evidence retrieval: The large language model generates search titles and search phrases based on the prompts, and retrieves external evidence based on the titles and search phrases through the search engine to obtain the text content of the evidence.
[0012] S3, knowledge graph representation of evidence, extracting entities of text evidence and the relationship between entities, and representing the retrieved text evidence in the form of triples to obtain knowledge graph triples;
[0013] S4. Reasoning and judgment: convert the knowledge graph triples into natural language understandable by the large language model. Use the large language model to filter triples that are useful for judging the truth of news. Input the triples that are useful for judging the truth as well as the image modality and text modality into the large language model to judge the truth of the news.
[0014] The forged information detection method combining bimodal understanding with a large language model as described above is characterized in that: in the term interpretation perspective of S1, the large language model generates interpretations of important words in the text information based on the text information and prompts of the given large language model.
[0015] The method for detecting forged information combining bimodal understanding and a large language model as described above is characterized in that: in the sentence perspective of S1, the large language model generates background knowledge of the sentence based on the prompt.
[0016] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that: in the multimodal perspective of S1, the large language model analyzes pairs of image information and text information according to prompts, generates and outputs the relationship between the image modality and the text modality.
[0017] The method for detecting forged information combining bimodal understanding with a large language model as described above is characterized in that when the image modality and text modality output by the large language model are unrelated, the being search engine is used to retrieve image evidence related to the text information.
[0018] The method for detecting forged information combining bimodal understanding with a large language model as described above is characterized in that: in S2, the being search engine is used to retrieve external evidence, and the text content of the evidence is obtained through the newspaper library in the programming language python.
[0019] The forged information detection method combining bimodal understanding and large language model as described above is characterized in that: S3 includes
[0020] S31, entity extraction, the large language model extracts and generates entity e according to the prompt i ;
[0021] S32, entity text summary, the large language model extracts the entity text from the prompti Extracting text summarization that matches the query or open-domain question answering tasks;
[0022] S33, entity mention acquisition, large language model identifies the text summary or open domain question answering task that needs to be analyzed based on the prompt In and entity e i Related entities j ;
[0023] S34, entity relationship extraction, text summarization or open domain question answering tasks according to the required analysis Entity i The set E i , and entity e i Related entities j , and the description of the relation extraction task, set the relation extraction prompt, and generate entity e i The relationship between , we get the knowledge graph triples.
[0024] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that the prompt in S31 includes an explanation of the entity recognition task, a description of the entity type, and a detailed description of the correct entity output format.
[0025] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that: the prompt in S33 includes a text summary or an open domain question answering task that needs to be analyzed and entity e i Related entities j The entity set E; the large language model in S33 rewrites the text summary or open domain question answering task that needs to be analyzed according to the prompt and select the entity e that is the same as the entity set E. j .
[0026] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that the large language model is a GPT large language model.
[0027] The beneficial effects of the present invention are:
[0028] When judging the truth or falsity of news, the present invention provides an explanation of the correlation between modes, which can better understand the relationship between modes, improve the accuracy of the large language model LLM in judging the truth or falsity of news, and avoid the problem of hallucination caused by the large language model LLM reasoning; at the same time, the retrieved text evidence is represented in the form of knowledge graph triples, so that the evidence is structured, which provides logic for reasoning and judgment; on the other hand, the advantages of the knowledge graph KG and the large language model LLM are fully utilized. The large language model LLM has a wide range of background knowledge, so the large language model LLM can assist in the extraction of entities and relationships of the knowledge graph KG, select information in the knowledge graph KG that is useful for judging the truth or falsity of news, and at the same time, the knowledge graph KG also provides structure and logic for the reasoning and judgment of the large language model LLM. [Drawings]
[0029] Figure 1 This is a schematic diagram of the present invention. [Specific implementation method]
[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.
[0031] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the descriptions of "preferred", "sub-preferred", etc. in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "preferred" or "sub-preferred" may explicitly or implicitly include at least one such feature.
[0032] like Figure 1 As shown in the figure, the forged information detection method combining bimodal understanding and large language model consists of four parts: multimodal understanding, evidence retrieval, knowledge graph representation of evidence, and reasoning judgment.
[0033] S1. Multimodal understanding. Multimodal understanding includes the perspective of noun interpretation, sentence perspective and multimodal perspective. It uses the multi-granularity thinking chain CoT method to explain the correlation between image modality and text modality through the large language model LLM, and judge whether the image is useful to the text.
[0034] Specifically, from the noun explanation perspective, given an LLM text information x, the prompt format is "Help me explain the meaning of special words for understanding in [x]", allowing the large language model LLM to generate explanations of important words in the text information.
[0035] The prompt input from the sentence perspective into the large language model LLM is "Explain the sentence to me with necessary background", allowing the large language model LLM to generate background knowledge of the sentence and provide rich semantic information for the text.
[0036] The multimodal perspective inputs an image and text pair into a large language model (LLM), where the image information is represented as I and the prompt input to the large language model (LLM) is “What is the relation between the [x] and the [I]?” The large language model (LLM) generates and outputs the relationship between the image modality and the text modality. When the output image modality and text modality are unrelated, the being search engine is used to retrieve image evidence related to the text information in the form of text queries and returned image results.
[0037] Among them, the multimodal understanding module can not only explain the detailed words and background semantics of the text modality, learn the relationship between the image modality, but also ensure the relevance of the text and image pairs, avoid irrelevant images, and provide explainability for subsequent reasoning and judgment.
[0038] S2. Evidence retrieval: First, the large language model LLM generates search titles and search phrases according to the prompts. Then, based on the titles and search phrases, the external evidence is retrieved through the being search engine. Finally, the text content of the evidence is obtained through the newspaper library in the programming language Python, that is, the text evidence E. v .
[0039] S3. Knowledge graph representation of evidence. The key to knowledge graph representation of evidence is to extract the entities of text evidence and the relationship between entities, and represent the retrieved text evidence in the form of triples, so that the retrieved evidence becomes structured and logical, which is convenient for the final reasoning and judgment large language model (LLM) to better understand the logical structure of the evidence and accurately judge the truth or falsity of the news.
[0040] How to correctly draw evidence from text v Extract entity e i , and accurately identifying the relationship between entities is the key to the knowledge graph representation of evidence. i, it is necessary to design appropriate entity extraction instructions for the large language model LLM, with a detailed description of entity types, entity extraction tasks and output formats. i Related Entities i , from the entire text evidence E v It is time-consuming and labor-intensive to search for entity e in the text, so it is necessary to use the large language model LLM to generate the entity e in the text according to the prompt. i Text summary Then, according to the prompt, let the large language model LLM summarize the text Find entity e in i Entity mentioned j Finally, according to the text summary Entity Set E i And each entity e learned i Entity mentioned j , as well as the description of the relation extraction task, design relation extraction prompts, generate relations between entities, and obtain knowledge graph triples.
[0041] Specifically, S3 includes
[0042] S31, entity extraction, using detailed system prompts to guide the large language model GPT-3.5 to generate entity e i The prompt includes an explanation of the entity recognition task, which is used to define the criteria for dividing entities in the large language model (LLM); provides a description of the entity type; and describes in detail the format of the correct entity output.
[0043] S32, entity text summary, large language model extracts entity e according to the prompt i Text summary And text summary It is consistent with the classic query-oriented abstract summarization or open-domain question answering task, where the open-domain question answering task can be performed again using the large language model GPT-3.5. i Extract information from simple declarative sentences.
[0044] S33, entity mention acquisition, in Find the references to e i Other entities that are relevant and contribute to its identity j To achieve this, the system prompt instructs the big voice model GPT-3.5 to identify the listed entities mentioned in a specific text, where the listed entities are provided by the user. The same list of entities is rewritten by the big language model GPT-3.5, with a "yes / no" answer at the end of each entry. The prompt provided by the user includes all entities e in the entity set E jA numbered list of generated text summaries or open-domain question answering tasks that need to be analyzed samples, and appropriately separate and label the entities e j , then select the entity marked as yes j sample.
[0045] S34, entity relationship extraction. Based on text summary Entity Set E i And each entity e learned i Entity mentioned j , as well as the description of the relation extraction task, design relation extraction prompts, generate relations between entities, and obtain knowledge graph triples.
[0046] S4. Reasoning and judgment: first convert the knowledge graph KG triples into natural language understandable by the large language model LLM, then use the large language model LLM to screen triples that are useful for judging the truth or falsity of the news, and finally input the triples that are useful for judging the truth or falsity as well as the image modality and text modality into the large language model LLM for the final judgment to judge the truth or falsity of the news.
[0047] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.
Claims
1. A method for detecting fake information by combining bimodal understanding with a large language model, characterized in that: Included S1. Multimodal understanding, including noun interpretation perspective, sentence perspective and multimodal perspective, uses multi-granularity thinking chains to explain the correlation between image modality and text modality through a large language model, and determines whether the image is useful for the text; S2. Evidence retrieval: The large language model generates search titles and search phrases based on the prompts, and retrieves external evidence based on the titles and search phrases through the search engine to obtain the text content of the evidence. S3, knowledge graph representation of evidence, extracting entities of text evidence and the relationship between entities, and representing the retrieved text evidence in the form of triples to obtain knowledge graph triples; S4. Reasoning and judgment: convert the knowledge graph triples into natural language understandable by the large language model. Use the large language model to filter triples that are useful for judging the truth of news. Input the triples that are useful for judging the truth as well as the image modality and text modality into the large language model to judge the truth of the news.
2. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S1’s noun interpretation perspective, the large language model generates interpretations of important words in the text information based on the text information and prompts given to the large language model.
3. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S1’s sentence perspective, the large language model generates background knowledge of the sentence based on the prompts.
4. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S1's multimodal perspective, the large language model analyzes pairs of image information and text information based on prompts, and generates and outputs the relationship between the image modality and the text modality.
5. The method for detecting forged information combining bimodal understanding and a large language model according to claim 4, characterized in that: When the large language model outputs that the image modality and text modality are unrelated, the being search engine is used to retrieve image evidence related to the text information.
6. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S2, the being search engine is used to retrieve external evidence, and the text content of the evidence is obtained through the newspaper library in the programming language Python.
7. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: S3 includes S31, entity extraction, the large language model extracts and generates entity e according to the prompt i ; S32, entity text summary, the large language model extracts the entity text from the prompt i Extracting text summarization that matches the query or open-domain question answering tasks; S33, entity mention acquisition, large language model identifies the text summary or open domain question answering task that needs to be analyzed based on the prompt In and entity e i Related entities j ; S34, entity relationship extraction, text summarization or open domain question answering tasks according to the required analysis Entity i The set E i , and entity e i Related entities j , and the description of the relation extraction task, set the relation extraction prompt, and generate entity e i The relationship between , we get the knowledge graph triples.
8. The method for detecting forged information combining bimodal understanding and a large language model according to claim 7, characterized in that: The prompts in S31 include an explanation of the entity recognition task, a description of the entity type, and a detailed description of the format of the correct entity output.
9. The method for detecting forged information combining bimodal understanding and a large language model according to claim 7, characterized in that: S33 prompts include text summarization or open-domain question answering tasks that need to be analyzed and entity e i Related entities j The entity set E; the large language model in S33 rewrites the text summary or open domain question answering task that needs to be analyzed according to the prompt and select the entity e that is the same as the entity set E. j .
10. The method for detecting forged information combining bimodal understanding and a large language model according to claim 7, characterized in that: The large language model is the GPT large language model.
Citation Information
Cited By
Zero-sample deep-forgery attribution method based on bimodal guidance
CN120220216A
Digital multimedia evidence obtaining method and device
CN121660104A
Multi-modal large model Deepfake detection method based on retrieval enhancement
CN121706005A
