Counterfeit information detection method combining bimodal understanding and large language model

By combining bimodal understanding and a fake information detection method of large language models, the problems of insufficient structural evidence and insufficient explanation of modal relationships in false news detection are solved, and more accurate judgments on the authenticity of news and logical reasoning are achieved. Reduced labor costs.

CN119003767APending Publication Date: 2024-11-22SHENZHEN KIM DAI INTELLIGENCE INNOVATION TECHNOLOGY CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410996900.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art has problems in the detection of false news, lack of structural evidence for retrieval, hallucination of model reasoning, and insufficient interpretation of modal relationships between images and texts, and building knowledge graphs requires a lot of labor costs and time.

Method used

Using a forged information detection method combining bimodal understanding and large language models, the modal correlation between image and text is explained through multi-grained thinking chains, search titles and phrases are generated to search evidence, extract entities and relationships and represent them as knowledge graph triplets , use large language models to make inference judgments.

Benefits of technology

Improve the interpretation of correlation between modals, enhance the accuracy of the judgment of true and false news, avoid model reasoning illusions, provide structured evidence representation, make full use of the advantages of knowledge graphs and large language models, and improve reasoning logic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003767A_ABST
    Figure CN119003767A_ABST
Patent Text Reader

Abstract

The invention discloses a counterfeited information detection method combining bimodal understanding and a large language model. The method comprises the following steps: S1, multimodal understanding; s2, evidence retrieval; s3, representing the knowledge graph of the evidence; s4, reasoning and judging; when the news is judged to be true or false, explanation of relevance between modes is provided, the relation between the modes can be better understood, the accuracy of judging the news to be true or false by a large language model is improved, the problem that illusion is generated by reasoning of the large language model is also avoided, meanwhile, retrieved text evidence is expressed in a knowledge graph triple form, and the news authenticity is judged to be true or false. The evidence is structured, and logicality is provided for reasoning and judgment.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a forged information detection method combining bimodal understanding with a large language model. [Background Technology]

[0002] As social media platforms are increasingly integrated into people's lives, they have become the main source of information for the public. Therefore, false news can be spread quickly, and the content of false news can easily confuse and mislead people, which in turn affects people's judgment and decision.

[0003] Multimodal large language models, such as GPT4 and llava, have powerful reasoning capabilities, zero-sample learning capabilities, and multimodal reasoning and judgment capabilities. They are widely used in forged information detection, but they still have the following problems:

[0004] 1. Although many studies on fake news detection use search engines to retrieve external evidence and use large language models (LLMs) to combine evidence with image-text modal reasoning and judgment, the retrieved evidence is mostly in the form of text and lacks structure, and the reasoning of large language models (LLMs) will produce hallucinations.

[0005] 2. Some studies on fake news detection based on large language models (LLMs) have failed to provide a reasonable explanation for the modal relationship between images and texts. Currently, many studies use contrastive learning methods, such as aligning image and text modalities, but the limitations of this method are long model training time, low efficiency, and lack of explainability of the relationship between modalities.

[0006] 3. The key to building a knowledge graph KG is to effectively extract entities and relationships, but identifying entities and the relationships between them requires knowledge of various fields of society, which requires a lot of manpower costs and is time-consuming. [Summary of the invention]

[0007] The present invention overcomes the deficiencies of the prior art and provides a method for detecting forged information that combines bimodal understanding with a large language model.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for detecting forged information combining bimodal understanding and a large language model, characterized in that:

[0010] S1. Multimodal understanding, including noun interpretation perspective, sentence perspective and multimodal perspective, uses multi-granularity thinking chains to explain the correlation between image modality and text modality through a large language model, and determines whether the image is useful for the text;

[0011] S2. Evidence retrieval: The large language model generates search titles and search phrases based on the prompts, and retrieves external evidence based on the titles and search phrases through the search engine to obtain the text content of the evidence.

[0012] S3, knowledge graph representation of evidence, extracting entities of text evidence and the relationship between entities, and representing the retrieved text evidence in the form of triples to obtain knowledge graph triples;

[0013] S4. Reasoning and judgment: convert the knowledge graph triples into natural language understandable by the large language model. Use the large language model to filter triples that are useful for judging the truth of news. Input the triples that are useful for judging the truth as well as the image modality and text modality into the large language model to judge the truth of the news.

[0014] The forged information detection method combining bimodal understanding with a large language model as described above is characterized in that: in the term interpretation perspective of S1, the large language model generates interpretations of important words in the text information based on the text information and prompts of the given large language model.

[0015] The method for detecting forged information combining bimodal understanding and a large language model as described above is characterized in that: in the sentence perspective of S1, the large language model generates background knowledge of the sentence based on the prompt.

[0016] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that: in the multimodal perspective of S1, the large language model analyzes pairs of image information and text information according to prompts, generates and outputs the relationship between the image modality and the text modality.

[0017] The method for detecting forged information combining bimodal understanding with a large language model as described above is characterized in that when the image modality and text modality output by the large language model are unrelated, the being search engine is used to retrieve image evidence related to the text information.

[0018] The method for detecting forged information combining bimodal understanding with a large language model as described above is characterized in that: in S2, the being search engine is used to retrieve external evidence, and the text content of the evidence is obtained through the newspaper library in the programming language python.

[0019] The forged information detection method combining bimodal understanding and large language model as described above is characterized in that: S3 includes

[0020] S31, entity extraction, the large language model extracts and generates entity e according to the prompt i ;

[0021] S32, entity text summary, the large language model extracts the entity text from the prompti Extracting text summarization that matches the query or open-domain question answering tasks;

[0022] S33, entity mention acquisition, large language model identifies the text summary or open domain question answering task that needs to be analyzed based on the prompt In and entity e i Related entities j ;

[0023] S34, entity relationship extraction, text summarization or open domain question answering tasks according to the required analysis Entity i The set E i , and entity e i Related entities j , and the description of the relation extraction task, set the relation extraction prompt, and generate entity e i The relationship between , we get the knowledge graph triples.

[0024] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that the prompt in S31 includes an explanation of the entity recognition task, a description of the entity type, and a detailed description of the correct entity output format.

[0025] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that: the prompt in S33 includes a text summary or an open domain question answering task that needs to be analyzed and entity e i Related entities j The entity set E; the large language model in S33 rewrites the text summary or open domain question answering task that needs to be analyzed according to the prompt and select the entity e that is the same as the entity set E. j .

[0026] The forged information detection method combining bimodal understanding and a large language model as described above is characterized in that the large language model is a GPT large language model.

[0027] The beneficial effects of the present invention are:

[0028] When judging the truth or falsity of news, the present invention provides an explanation of the correlation between modes, which can better understand the relationship between modes, improve the accuracy of the large language model LLM in judging the truth or falsity of news, and avoid the problem of hallucination caused by the large language model LLM reasoning; at the same time, the retrieved text evidence is represented in the form of knowledge graph triples, so that the evidence is structured, which provides logic for reasoning and judgment; on the other hand, the advantages of the knowledge graph KG and the large language model LLM are fully utilized. The large language model LLM has a wide range of background knowledge, so the large language model LLM can assist in the extraction of entities and relationships of the knowledge graph KG, select information in the knowledge graph KG that is useful for judging the truth or falsity of news, and at the same time, the knowledge graph KG also provides structure and logic for the reasoning and judgment of the large language model LLM. [Drawings]

[0029] Figure 1 This is a schematic diagram of the present invention. [Specific implementation method]

[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.

[0031] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the descriptions of "preferred", "sub-preferred", etc. in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "preferred" or "sub-preferred" may explicitly or implicitly include at least one such feature.

[0032] like Figure 1 As shown in the figure, the forged information detection method combining bimodal understanding and large language model consists of four parts: multimodal understanding, evidence retrieval, knowledge graph representation of evidence, and reasoning judgment.

[0033] S1. Multimodal understanding. Multimodal understanding includes the perspective of noun interpretation, sentence perspective and multimodal perspective. It uses the multi-granularity thinking chain CoT method to explain the correlation between image modality and text modality through the large language model LLM, and judge whether the image is useful to the text.

[0034] Specifically, from the noun explanation perspective, given an LLM text information x, the prompt format is "Help me explain the meaning of special words for understanding in [x]", allowing the large language model LLM to generate explanations of important words in the text information.

[0035] The prompt input from the sentence perspective into the large language model LLM is "Explain the sentence to me with necessary background", allowing the large language model LLM to generate background knowledge of the sentence and provide rich semantic information for the text.

[0036] The multimodal perspective inputs an image and text pair into a large language model (LLM), where the image information is represented as I and the prompt input to the large language model (LLM) is “What is the relation between the [x] and the [I]?” The large language model (LLM) generates and outputs the relationship between the image modality and the text modality. When the output image modality and text modality are unrelated, the being search engine is used to retrieve image evidence related to the text information in the form of text queries and returned image results.

[0037] Among them, the multimodal understanding module can not only explain the detailed words and background semantics of the text modality, learn the relationship between the image modality, but also ensure the relevance of the text and image pairs, avoid irrelevant images, and provide explainability for subsequent reasoning and judgment.

[0038] S2. Evidence retrieval: First, the large language model LLM generates search titles and search phrases according to the prompts. Then, based on the titles and search phrases, the external evidence is retrieved through the being search engine. Finally, the text content of the evidence is obtained through the newspaper library in the programming language Python, that is, the text evidence E. v .

[0039] S3. Knowledge graph representation of evidence. The key to knowledge graph representation of evidence is to extract the entities of text evidence and the relationship between entities, and represent the retrieved text evidence in the form of triples, so that the retrieved evidence becomes structured and logical, which is convenient for the final reasoning and judgment large language model (LLM) to better understand the logical structure of the evidence and accurately judge the truth or falsity of the news.

[0040] How to correctly draw evidence from text v Extract entity e i , and accurately identifying the relationship between entities is the key to the knowledge graph representation of evidence. i, it is necessary to design appropriate entity extraction instructions for the large language model LLM, with a detailed description of entity types, entity extraction tasks and output formats. i Related Entities i , from the entire text evidence E v It is time-consuming and labor-intensive to search for entity e in the text, so it is necessary to use the large language model LLM to generate the entity e in the text according to the prompt. i Text summary Then, according to the prompt, let the large language model LLM summarize the text Find entity e in i Entity mentioned j Finally, according to the text summary Entity Set E i And each entity e learned i Entity mentioned j , as well as the description of the relation extraction task, design relation extraction prompts, generate relations between entities, and obtain knowledge graph triples.

[0041] Specifically, S3 includes

[0042] S31, entity extraction, using detailed system prompts to guide the large language model GPT-3.5 to generate entity e i The prompt includes an explanation of the entity recognition task, which is used to define the criteria for dividing entities in the large language model (LLM); provides a description of the entity type; and describes in detail the format of the correct entity output.

[0043] S32, entity text summary, large language model extracts entity e according to the prompt i Text summary And text summary It is consistent with the classic query-oriented abstract summarization or open-domain question answering task, where the open-domain question answering task can be performed again using the large language model GPT-3.5. i Extract information from simple declarative sentences.

[0044] S33, entity mention acquisition, in Find the references to e i Other entities that are relevant and contribute to its identity j To achieve this, the system prompt instructs the big voice model GPT-3.5 to identify the listed entities mentioned in a specific text, where the listed entities are provided by the user. The same list of entities is rewritten by the big language model GPT-3.5, with a "yes / no" answer at the end of each entry. The prompt provided by the user includes all entities e in the entity set E jA numbered list of generated text summaries or open-domain question answering tasks that need to be analyzed samples, and appropriately separate and label the entities e j , then select the entity marked as yes j sample.

[0045] S34, entity relationship extraction. Based on text summary Entity Set E i And each entity e learned i Entity mentioned j , as well as the description of the relation extraction task, design relation extraction prompts, generate relations between entities, and obtain knowledge graph triples.

[0046] S4. Reasoning and judgment: first convert the knowledge graph KG triples into natural language understandable by the large language model LLM, then use the large language model LLM to screen triples that are useful for judging the truth or falsity of the news, and finally input the triples that are useful for judging the truth or falsity as well as the image modality and text modality into the large language model LLM for the final judgment to judge the truth or falsity of the news.

[0047] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A method for detecting fake information by combining bimodal understanding with a large language model, characterized in that: Included S1. Multimodal understanding, including noun interpretation perspective, sentence perspective and multimodal perspective, uses multi-granularity thinking chains to explain the correlation between image modality and text modality through a large language model, and determines whether the image is useful for the text; S2. Evidence retrieval: The large language model generates search titles and search phrases based on the prompts, and retrieves external evidence based on the titles and search phrases through the search engine to obtain the text content of the evidence. S3, knowledge graph representation of evidence, extracting entities of text evidence and the relationship between entities, and representing the retrieved text evidence in the form of triples to obtain knowledge graph triples; S4. Reasoning and judgment: convert the knowledge graph triples into natural language understandable by the large language model. Use the large language model to filter triples that are useful for judging the truth of news. Input the triples that are useful for judging the truth as well as the image modality and text modality into the large language model to judge the truth of the news.

2. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S1’s noun interpretation perspective, the large language model generates interpretations of important words in the text information based on the text information and prompts given to the large language model.

3. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S1’s sentence perspective, the large language model generates background knowledge of the sentence based on the prompts.

4. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S1's multimodal perspective, the large language model analyzes pairs of image information and text information based on prompts, and generates and outputs the relationship between the image modality and the text modality.

5. The method for detecting forged information combining bimodal understanding and a large language model according to claim 4, characterized in that: When the large language model outputs that the image modality and text modality are unrelated, the being search engine is used to retrieve image evidence related to the text information.

6. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: In S2, the being search engine is used to retrieve external evidence, and the text content of the evidence is obtained through the newspaper library in the programming language Python.

7. The method for detecting forged information combining bimodal understanding and a large language model according to claim 1, characterized in that: S3 includes S31, entity extraction, the large language model extracts and generates entity e according to the prompt i ; S32, entity text summary, the large language model extracts the entity text from the prompt i Extracting text summarization that matches the query or open-domain question answering tasks; S33, entity mention acquisition, large language model identifies the text summary or open domain question answering task that needs to be analyzed based on the prompt In and entity e i Related entities j ; S34, entity relationship extraction, text summarization or open domain question answering tasks according to the required analysis Entity i The set E i , and entity e i Related entities j , and the description of the relation extraction task, set the relation extraction prompt, and generate entity e i The relationship between , we get the knowledge graph triples.

8. The method for detecting forged information combining bimodal understanding and a large language model according to claim 7, characterized in that: The prompts in S31 include an explanation of the entity recognition task, a description of the entity type, and a detailed description of the format of the correct entity output.

9. The method for detecting forged information combining bimodal understanding and a large language model according to claim 7, characterized in that: S33 prompts include text summarization or open-domain question answering tasks that need to be analyzed and entity e i Related entities j The entity set E; the large language model in S33 rewrites the text summary or open domain question answering task that needs to be analyzed according to the prompt and select the entity e that is the same as the entity set E. j .

10. The method for detecting forged information combining bimodal understanding and a large language model according to claim 7, characterized in that: The large language model is the GPT large language model.

Citation Information

Cited By

  • Zero-sample deep-forgery attribution method based on bimodal guidance

    CN120220216A

  • Digital multimedia evidence obtaining method and device

    CN121660104A

  • Multi-modal large model Deepfake detection method based on retrieval enhancement

    CN121706005A