Multi-mode error information detection method and system of mixed source

Abstract

The invention provides a mixed-source multi-mode error information detection method and system, and belongs to the technical field of error information detection.The method comprises the steps that target information is obtained; inputting the target information into a target detection model, and judging whether the target information is error information or not; the target detection model is a multi-modal error information detection model obtained based on multi-class error information training, and the information sources of the multi-class error information are different. According to the multi-mode error information detection method and system based on the mixed sources, the adaptability of the detection model to complex error types can be effectively improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of error information detection, and more specifically, to a mixed-source multimodal error information detection method and system. Background Art

[0002] With the continuous development and widespread application of generative models, the cost of producing various types of false information has gradually decreased, posing a huge threat to multiple fields such as finance and public health, and triggering serious concerns about the authenticity of information in society and the public. Therefore, it is urgent to identify multimodal misinformation on social media. Existing multimodal misinformation detection methods usually assume that each sample has a known single source of forgery. This assumption is too simple and cannot capture the complexity of real-world scenarios, because in the real world, misinformation often comes from multiple forgery sources. Therefore, existing datasets mainly consist of misinformation from a single source, and there is a lack of multimodal misinformation datasets from mixed sources. How to improve the adaptability of detection models to complex error types is a problem that needs to be solved at present. Summary of the invention

[0003] The purpose of the present disclosure is to provide a mixed-source multimodal error information detection method and system to break through the traditional single-source limitation and effectively improve the adaptability of the detection model to complex error types.

[0004] According to a first aspect of an embodiment of the present disclosure, a mixed-source multimodal error information detection method is provided, comprising: Get target information; The target information is input into a target detection model to determine whether the target information is error information; the target detection model is a multimodal error information detection model trained based on multiple types of error information, and the multiple types of error information have different information sources.

[0005] A second aspect of the embodiments of the present disclosure provides a mixed-source multimodal error information detection system, including: Information module, used to obtain target information; The detection module is used to input the target information into a target detection model to determine whether the target information is error information; the target detection model is a multimodal error information detection model trained based on multiple types of error information, and the information sources of the multiple types of error information are different.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned mixed-source multimodal error information detection method when executing the computer program.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned mixed-source multimodal error information detection method are implemented.

[0008] The mixed-source multi-modal error information detection method and system provided by the embodiments of the present disclosure have the following beneficial effects: The disclosed embodiment integrates multiple sources of error information to build a powerful multimodal detection model, namely a target detection model, which not only covers multiple information modalities, but also effectively identifies potential errors across modalities, significantly enhancing the adaptability of error information identification. In addition, the disclosed embodiment breaks through the traditional single-source limitation and promotes the development of real-world error information identification tasks. Therefore, the disclosed embodiment breaks through the traditional single-source limitation and effectively improves the adaptability of the detection model to complex error types. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of a flow chart of a mixed-source multi-modal error information detection method provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of forgery types and data ratios of multi-modal mixed-source forged information and using a generative model and artificial intelligence tools to generate erroneous information from different sources provided by an embodiment of the present disclosure; Figure 3 A structural block diagram of the overall framework of MMD-Agent provided in one embodiment of the present disclosure; Figure 4 A structural block diagram of a mixed-source multi-modal error information detection system provided by an embodiment of the present disclosure; Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0012] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0013] Please refer to Figure 1 , Figure 1 A flow chart of a mixed-source multi-modal error information detection method provided by an embodiment of the present disclosure, the method comprising: S101: Acquire target information.

[0014] In this embodiment, the target information is information to be detected, including multiple types of information (ie, true information and false information, wherein the types of false information include multiple types), and the information sources are different.

[0015] Specifically, the target information can be obtained as follows: First, clarify the information needs so that you can search in a targeted manner. Clarifying the subject, scope and key points of the information will help you quickly filter out relevant content from the massive amount of information.

[0016] Second, collect information through multiple channels.

[0017] Online databases and search engines: Use professional databases, such as academic databases, industry databases, etc.; well-known search engines can quickly find a large amount of relevant information.

[0018] Social media platforms: Follow experts, institutions, and practitioners in related fields who will share valuable information and opinions; participate in discussions on related topics and gain information from different people’s perspectives.

[0019] News media: Subscribe to professional news media to learn about industry trends and latest events; pay attention to different types of media, including traditional media and new media, to obtain a more comprehensive information perspective.

[0020] S102: Input the target information into a target detection model to determine whether the target information is error information; the target detection model is a multimodal error information detection model trained based on multiple types of error information, and the multiple types of error information have different information sources.

[0021] In this embodiment, the target detection model is responsible for receiving target information in multimodal error information detection, and determining whether the above information is error information through analysis and judgment.

[0022] The object detection model is trained based on multiple types of error information. At this point, the model has learned various types of error patterns and features. During the training process, the object detection model will learn multiple types of error information from different information sources, so that it can identify a wide range of error types.

[0023] For example, for text information, the object detection model can learn error patterns such as grammatical errors, logical contradictions, and false statements; for image information, the object detection model can identify traces of tampering, unnatural color distribution, abnormal object shapes, etc.

[0024] The information sources of multiple types of error messages are different, providing rich learning information for the target detection model. Different information sources can include the following: First, the news media.

[0025] News reports may contain false information, such as inaccurate facts, exaggeration, one-sided reporting, etc. News media sources of information are wide-ranging, covering various fields such as economy, society, and technology, so the types of false information are also varied.

[0026] For example, a news report about an event may contain errors in time, place, characters, etc., or the description of the event may be biased.

[0027] Second, social media.

[0028] User-generated content on social media platforms is a common source of misinformation. Users may post false information, rumors, misleading content, etc. for various purposes.

[0029] For example, unconfirmed rumors, false health advice, and exaggerated descriptions of events spread on social media.

[0030] Third, the source of images and videos.

[0031] Images and videos can be tampered with, synthesized or edited to produce false information, such as photoshopped images, synthesized videos, etc. Some false advertising campaigns use tampered images or videos to mislead consumers.

[0032] Specifically, in order for the target detection model to accurately determine whether the target information is false information, a lot of training and optimization is required. The process is as follows: First, information collection and labeling.

[0033] Collect multiple types of erroneous information (historical information) from different information sources, and label the above information, clearly indicate which is the erroneous information and what the type of error is, and ensure the accuracy and consistency of the labeling.

[0034] Second, model training.

[0035] The labeled multiple types of error messages are input into the detection model for training. The detection model will gradually improve its ability to identify error messages by learning the features and patterns of the above error messages.

[0036] During the training process, different optimization algorithms and loss functions can be used to improve the performance and convergence speed of the model.

[0037] Third, detection model evaluation and optimization.

[0038] Use the validation set to evaluate the trained detection model and check the F1 value, macro precision, macro recall, and macro accuracy of the detection model.

[0039] According to the evaluation results, the detection model is further optimized, such as adjusting the model parameters, adding training data, improving the model structure, etc., to obtain the final target detection model.

[0040] From the above, it can be concluded that this embodiment integrates multiple types of error information sources to build a powerful multimodal detection model, namely a target detection model, which not only covers multiple information modalities, but also effectively identifies potential errors across modalities, significantly enhancing the adaptability of error information identification. In addition, this embodiment breaks through the traditional single-source limitation and promotes the development of real-world error information identification tasks. Therefore, this embodiment breaks through the traditional single-source limitation and effectively improves the adaptability of the detection model to complex error types.

[0041] In one embodiment of the present disclosure, the mixed-source multimodal error information detection method further includes: Get real information; Obtain multiple types of error information from different sources; A first sample data set is established based on real information and multiple types of false information; Among them, the first sample data set is the MMFakeBench data set.

[0042] Specifically, this embodiment collects real information (3,300 pairs of real data pairs), ensures the accuracy of text and images, and shows strong image-text consistency. Since the synthetic data of this embodiment comes from multiple datasets, a real dataset can be constructed from the same corresponding sources, including MS-COCO, VisualNews, and real image-text pairs from Fakeddit. Among them, the real data all comes from existing datasets, and the synthetic data refers to data combined from multiple different dataset sources. Then, this embodiment further divides VisualNews into four different news sources: The Guardian, BBC, USA Today, and The Washington Post. Finally, this embodiment constructs a real dataset by selecting on average from 6 different sources, that is, 550 pairs of real data pairs are randomly selected from each source, and the number of data from each data source is averaged.

[0043] This embodiment also collects multiple types of error information. Afterwards, the real information and the multiple types of error information are integrated to establish a first sample data set (ie, MMFakeBench data set).

[0044] Among them, the MMFakeBench dataset can be used to train and evaluate various multimodal false information detection models. By training on this dataset, the detection model can learn different types of false information features and patterns, and improve the ability to identify false information. At the same time, this dataset can also be used as a benchmark dataset to compare the performance and effects of different false information detection methods, and promote the development of multimodal false information detection technology.

[0045] It can be concluded from the above that this embodiment ensures the diversity and authenticity of the data set, and effectively simulates the complex and changeable false information environment in the real world by integrating real information with multi-source false information. At the same time, the data set provides researchers with rich experimental data, which helps to develop more accurate and robust false information detection models and improve the accuracy and efficiency of information identification.

[0046] In one embodiment of the present disclosure, the mixed-source multimodal error information detection method further includes: The object detection model is trained based on a large visual language model and the first sample dataset.

[0047] In this embodiment, the large language model has a strong language understanding ability and can perform in-depth analysis of text information. In multimodal error information detection, for target information containing a large amount of text description, the large language model can extract key semantic information, identify grammatical structures and logical relationships.

[0048] For example, for a news report, the large language model can analyze the article's theme, viewpoints, event development context, etc., providing an important basis for judging the accuracy of the information.

[0049] The Large Vision-Language Model (LVLM) is derived from the Large Language Model. This embodiment extends the ability to perceive visual signals to the Large Language Model to form the LVLM. LVLM refers to a deep learning model with large-scale parameters that can simultaneously understand the semantic content of images and natural language. The above model combines powerful processing capabilities in the fields of computer vision and natural language processing, and is typically used to perform tasks between vision and language. Therefore, LVLM inherits the large language model's ability to recognize the open world and certain reasoning capabilities. At present, there are two main formation principles of LVLM: (1) Feature alignment-based method: First, image-text feature alignment is performed on the large language model, and then the visual instructions are fine-tuned to enable the large language model to have cross-modal capabilities.

[0050] (2) Methods based on multimodal input. By directly building a simple linear layer between the large language model and the visual model, the model can directly allow multimodal input. When using LVLM, given a carefully designed prompt word, LVLM will output the answer based on the embedding and prompt word requirements.

[0051] LVLM's ability to support multimodal input and powerful open-world cognitive capabilities provide an important methodological basis for multimodal misinformation detection.

[0052] LVLM has been trained with a large amount of data and has rich knowledge representation. When training the object detection model, LVLM can use its knowledge reserve for reasoning, helping the model to better understand the context and meaning of the information.

[0053] For example, when encountering an error message involving knowledge in a specific domain, LVLM can provide the model with relevant knowledge and reasoning clues based on its understanding of the domain, thereby improving the accuracy of error message detection.

[0054] The MMFakeBench dataset (the first sample dataset) contains multiple categories of false information from different information sources, covering various error types such as false statements, exaggerations, misleading information, tampered content, etc., so that the trained target detection model can be exposed to a wide range of error patterns and improve the ability to identify different types of false information.

[0055] The first sample dataset contains not only false information but also true information. By comparing and learning true information with false information, the target detection model can better understand the characteristics and patterns of the correct information, thereby more accurately identifying false information.

[0056] For example, the target detection model can learn the language style, logical structure, and factual basis of real news reports, and then compare them with fake news to find the differences between the two and improve the ability to detect fake news.

[0057] The architecture of the target detection model is built based on LVLM, and specific modules and layers are added on its basis to adapt to the task requirements of multimodal misinformation detection.

[0058] For example, in this embodiment, a classifier can be added to the output layer of LVLM to determine whether the input information is erroneous information; or an attention mechanism module can be added to better integrate multimodal information.

[0059] During the training process, the model needs to be evaluated to understand the performance and progress of the model. The model can be evaluated using the validation set to calculate the F1 value, macro precision, macro recall, and macro accuracy. Based on the evaluation results, the model can be adjusted and optimized.

[0060] After training based on LVLM and the first sample dataset, the target detection model can effectively fuse multimodal information and comprehensively utilize the features of different modalities such as text and images to judge the accuracy of information.

[0061] Specifically, the MMFakeBench dataset consists of 11,000 image-text pairs, which are divided into a validation set and a test set. The validation set contains 1,000 image-text pairs for hyperparameter selection, while the test set contains 10,000 pairs. The MMFakeBench dataset includes one true category and three false information categories. Detailed statistics are shown in Figure 2. Figure 2 (a) shows. MMFakeBench is divided into 30% text authenticity distortion, 10% visual authenticity distortion, 10% cross-modal consistency distortion, and 30% real data. According to the source of text and image, the above three error information categories can be further subdivided into 12 detailed subcategories, such as Figure 2 Such a comprehensive benchmark highlights the challenges of dealing with mixed sources and multiple types of multimodal misinformation intertwined in the real world.

[0062] For example, Figure 2(a) shows the types and proportions of forged information from multimodal mixed sources. Among them, 12.5% ​​are reused images (7.5% are natural rumors, 5% are rumors generated by GPT); 17.5% are AI-generated images (2.5% are natural rumors, 10% are artificial rumors, and 5% are rumors generated by GPT); 5% are AI-generated images; 5% are PS-edited images; 15% are reused inconsistencies (5% are person queries, 5% are semantic queries, and 5% are scene queries); 15% are edited inconsistencies (10% are text edits and 5% are image edits).

[0063] Figure 2 (b)-(d) illustrate the use of generative models and AI tools to generate misinformation from different sources, where: Figure 2 (b) shows text authenticity distortion (i.e., text misinformation) using AI-generated image pairings; Figure 2 (c) Demonstrates distortions in visual reality (i.e., visual misinformation) using AI-generated images; Figure 2 (d) shows the cross-modal consistency distortion of text / image editing type (i.e., cross-modal consistency misinformation).

[0064] This example proposes a simple yet effective framework, called MMD-Agent, which integrates the reasoning, action, and tool-use capabilities of LVLM. Figure 3 As shown in Figure 1, MMD-Agent includes two main processes: hierarchical decomposition and multi-perspective principle integration. Specifically, this embodiment instructs LVLM to decompose the mixed-source multimodal misinformation detection task into three smaller subtasks: text authenticity check, visual authenticity check, and cross-modal consistency reasoning. In the intermediate stage, each subtask t generates a multi-perspective reasoning and compare it with the model action Combined to facilitate decision making :

[0065] in, represents decision making, M() represents LVLM, Represents model actions, Indicates multi-angle reasoning.

[0066] This embodiment provides valuable insights by reasoning in the current environment, thereby meeting the requirement of finding different clues from error messages. Figure 3As shown in (c), the above reasons can include multiple perspectives, such as text key entities (reasoning principle 1), injected retrieval knowledge (reasoning principle 2), fact analysis (reasoning principle 3), and common sense analysis (reasoning principles 5 and 7). Under the guidance of the model, using its inherent understanding and capabilities, it generates reasoning paths and makes decisions for each subtask.

[0067] For example, Figure 3 (a) shows mixed-source multimodal error information. Fine-grained classification of multimodal error information makes accurate judgments.

[0068] Figure 3 (b) The standard prompt words are shown. For example, follow the instructions below: (1) Is there any objective evidence to refute the news headline? If yes, answer “Complete [textual distortion].” If no, proceed to step (2); (2) Is there any objective evidence to refute this news image? If yes, answer “Complete [Visual Reality Distortion].” If no, proceed to step (3); (3) Do the news headlines match the news pictures? If yes, answer “Complete [Original picture and text match].” If no, answer “Complete [Picture and text do not match].”

[0069] Therefore, the answer to Example 1 is: complete [the picture and text do not match]; the answer to Example 2 is: complete [the original picture and text are matched]; the answer to Example 3 is: complete [the original picture and text are matched].

[0070] Figure 3 (c) shows the MMD agent.

[0071] First, perform text authenticity check.

[0072] Question: Is the authenticity of the text distorted? Behavior 1: Find key entities; Behavior 3: Analyze whether there are objective facts to support or refute the news headline; Behavior 4: Draw conclusions.

[0073] Then, a visual authenticity check is performed.

[0074] Question: Is visual reality distorted? Behavior 5: Describing content in news images that violates objective facts; Behavior 6: Draw conclusions.

[0075] Finally, cross-modal consistency reasoning is performed.

[0076] Question: Is judging cross-modal authenticity distorted? Behavior 7: Draw conclusions.

[0077] From the above, it can be concluded that the MMFakeBench dataset (the first sample dataset) can provide sufficient training data for the target detection model, so that the target detection model can learn a wider range of information features and patterns and improve the generalization ability of the model. At the same time, through the powerful understanding and generalization ability of LVLM, the model in this embodiment can understand the contextual information in the image at a deeper level and improve the accuracy of detection.

[0078] In one embodiment of the present disclosure, the multiple types of error messages include text error messages; Getting multiple types of error information also includes: Get text rumors; Identify text misinformation based on text rumors.

[0079] In this embodiment, text misinformation can be text authenticity distortion. Text authenticity distortion is a key category of misinformation. Previous work focused on single-source text rumors, but the types of rumors in the real world are diverse, and text rumors accompanied by images will have a greater impact. To solve the above problems, MMFakeBench introduces three types of rumors: (1) Natural rumors: This example collects natural rumors from two websites, Politifact and Gossipcop, which provide news and entertainment stories.

[0080] (2) Artificial rumors: This example collects artificial rumors from the Fever dataset, which is constructed by manually modifying sentences from Wikipedia.

[0081] (3) Rumors generated by GPT: This embodiment instructs GPT to generate rumors through three methods: arbitrary generation, rewriting rumors, and information tampering. In order to increase the credibility of the rumors, this embodiment uses images that are highly relevant to the content of the rumors for pairing. The sources of the images are twofold: 1) AI-generated images. Figure 2 As shown in (b), this embodiment uses an image generation model (Stable Diffusion XL, DALL-E3, Midjourney V6) based on the rumor content to generate an image that is highly consistent with the rumor text description. 2) Reuse of real images. This embodiment selects the real image that is closest to the rumor description for pairing based on the text-image CLIP similarity score and the text-text CLIP similarity score.

[0082] It can be concluded from the above that this embodiment increases the diversity of types of text error information (i.e., text authenticity distortion).

[0083] In one embodiment of the present disclosure, the multiple types of error information include visual error information; Getting multiple types of error information also includes: The text error information is forged to obtain the visual error information corresponding to the text error information.

[0084] In this embodiment, the visual error information may be a visual authenticity distortion. Visual authenticity distortion refers to the situation where the text in a text-image pair is real but the image is erroneous. Based on the technology of forging erroneous images, MMFakeBench introduces two types of visual authenticity distortion: (1) PS edited images: PS edited images come from the Fakeeddit dataset, from which this embodiment manually selects highly harmful images as samples.

[0085] (2) AI-generated images: Figure 2 As shown in (c), this embodiment first selects real text-image pairs from the MS-COCO dataset and the VisualNews dataset, and then uses ChatGPT to generate corresponding factual conflict descriptions based on the original text title. Finally, this embodiment combines the original text title and the factual conflict description as prompt words to guide the generation model to generate corresponding images.

[0086] It can be concluded from the above that this embodiment increases the richness of visual information errors (ie, visual authenticity distortion) by using the technology of forging erroneous images.

[0087] In one embodiment of the present disclosure, the multiple types of error information include cross-modality consistency error information; Getting multiple types of error information also includes: Cross-modal consistency error information is obtained based on inconsistency transformation.

[0088] In this embodiment, the cross-modal consistency error information can be a cross-modal consistency distortion. In the cross-modal consistency distortion, both the text and the image are accurate, but either the text or the image is replaced and tampered with to destroy their overall consistency. Previous datasets focus on inconsistencies from a single source. In contrast, MMFakeBench integrates inconsistencies from the perspectives of editing and reuse into 5 different subcategories. 1) Re-adjustment inconsistency. The dataset of this embodiment includes three types of reuse inconsistencies, semantic queries, person queries, and scene queries. Specifically, semantic queries retrieve reused images based on specific semantic content, person queries ensure that the person mentioned in the title appears in the mismatched image, and scene queries rely on spatial similarity to retrieve similar but inconsistent scene information from the reused image. 2) Editing inconsistency. The dataset of this embodiment contains two types of editing inconsistencies: text editing and image editing. For text editing, this embodiment selects samples from DGM4 and COCO-Counterfactuals datasets. As Figure 2 As shown in (d), for image editing, this embodiment extracts the word differences between the original text and the edited text from the COCO-Counterfactuals dataset, and selects samples with significant semantic differences using CLIP similarity.

[0089] It can be concluded from the above that this embodiment enriches the dataset of error information by subdividing the types of cross-modal consistent error information (i.e., cross-modal consistent distortion) and extracting or selecting samples accordingly.

[0090] In one embodiment of the present disclosure, the mixed-source multimodal error information detection method further includes: The target detection model is improved based on retrieval enhancement and tool learning to obtain an improved target detection model.

[0091] Specifically, if is the multimodal sample pair to be verified, is an external knowledge base that covers many text and image contents. The retrieval enhancement technology extracts features from the text and image in the multimodal sample, and then retrieves relevant matching cases from the external knowledge base. Define a retriever, , the retriever obtains Search and Related information. It can be implemented as a retrieval system based on similarity measurement, where similarity express The data points in the input image and text are The correlation between:

[0092] Among them, the similarity can be calculated by cosine similarity, Euclidean distance or feature extraction model based on deep learning.

[0093] After retrieving external knowledge, LVLM takes multimodal data pairs and external knowledge as input, and analyzes whether there is a conflict between data information and external knowledge according to the hierarchical task decoupling step. In task decoupling, the complex multimodal false information detection problem of this framework is refined into subtasks such as text authenticity check, visual authenticity check, and cross-modal consistency reasoning. Then, by guiding the model to generate multi-perspective reasoning paths, these reasoning paths include text common sense analysis, image counterfactual content analysis, etc., to enhance the accuracy of decision-making. Combined with the integration of multi-perspective principles, LVLM finally outputs the specific error type of the data.

[0094] The new framework also includes tool learning techniques that enable the model to identify and adopt the most effective tools and strategies in actual operations. For example, when a specific type of multimodal misinformation is detected, the model first infers whether there is enough evidence to prove that the information is misinformation or correct information based on the existing text and images. When there is not enough evidence, the model will call on tools, such as Google search, to find more external information to help the model make a judgment. For a set of possible tools, each tool Each has its own specific application conditions and effectiveness. Define a decision function , which selects the optimal tool based on the current system state (composed of the data and the current state of the model).

[0095]

[0096] in, Indicates in status Next, use the tool effectiveness. It refers to the current state of the model, that is, whether the model can judge whether the information is wrong or not based on existing knowledge.

[0097] It can be concluded from the above that in order to further enhance the functionality of MMD-Agent, the new framework of this embodiment integrates advanced retrieval enhancement technology, which enables the model to dynamically retrieve relevant information and cases from a wide range of data sources before performing tasks. The above method not only improves the adaptability of the model to new and complex error types, but also enhances its responsiveness when facing unknown situations. This embodiment also introduces tool learning technology to enable the model to identify and adopt the most effective tools and strategies in actual operations.

[0098] Corresponding to the mixed-source multimodal error information detection method of the above embodiment, Figure 4A structural block diagram of a mixed-source multi-modal error information detection system provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The mixed-source multimodal error information detection system 20 includes: an information module 21 and a detection module 22 .

[0099] Wherein, the information module 21 is used to obtain target information; The detection module 22 is used to input the target information into the target detection model to determine whether the target information is error information; the target detection model is a multimodal error information detection model trained based on multiple types of error information, and the information sources of the multiple types of error information are different.

[0100] In one embodiment of the present disclosure, the mixed-source multimodal error information detection system 20 further includes: a data module; Data module, used to obtain real information; Obtain multiple types of error information from different sources; A first sample data set is established based on real information and multiple types of false information; Among them, the first sample data set is the MMFakeBench data set.

[0101] In one embodiment of the present disclosure, the mixed-source multimodal false information detection system 20 further includes: a training module; The training module is used to train an object detection model based on a large visual language model and a first sample data set.

[0102] In one embodiment of the present disclosure, the data module is specifically used for: Multiple categories of error messages include text error messages; Getting multiple types of error information also includes: Get text rumors; Identify text misinformation based on text rumors.

[0103] In one embodiment of the present disclosure, the data module is specifically used for: Multiple categories of misinformation include visual misinformation; Getting multiple types of error information also includes: The text error information is forged to obtain the visual error information corresponding to the text error information.

[0104] In one embodiment of the present disclosure, the data module is specifically used for: Multiple types of misinformation include cross-modal consistency misinformation; Getting multiple types of error information also includes: Cross-modal consistency error information is obtained based on inconsistency transformation.

[0105] In one embodiment of the present disclosure, the mixed-source multimodal error information detection system 20 further includes: an improvement module; The improvement module is used to improve the target detection model based on retrieval enhancement and tool learning to obtain an improved target detection model.

[0106] See also Figure 5 , Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 5 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned system embodiments, such as Figure 4 The functions of modules 21 to 22 are shown.

[0107] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0109] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0110] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure may execute the implementation methods described in the first and second embodiments of the mixed-source multimodal error information detection method provided in the embodiments of the present disclosure, and may also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0111] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or system capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0112] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0113] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0117] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0118] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A mixed-source multimodal error information detection method, characterized in that: include: Get target information; Inputting the target information into a target detection model to determine whether the target information is error information; The target detection model is a multimodal error information detection model trained based on multiple types of error information, and the multiple types of error information have different information sources.

2. The mixed-source multimodal error information detection method according to claim 1, characterized in that: Also includes: Get real information; Acquire the multiple types of error information, where the multiple types of error information come from different sources; Establishing a first sample data set according to the true information and the multiple types of false information; Among them, the first sample data set is the MMFakeBench data set.

3. The mixed-source multimodal error information detection method according to claim 2, characterized in that: Also includes: The object detection model is obtained by training based on a large visual language model and the first sample data set.

4. The mixed-source multimodal error information detection method according to claim 2, characterized in that: The multiple types of error messages include text error messages; The obtaining of the multiple types of error information further includes: Get text rumors; The text false information is determined according to the text rumor.

5. The mixed-source multimodal error information detection method according to claim 4, characterized in that: The multiple types of error messages include visual error messages; The obtaining of the multiple types of error information further includes: The text error information is forged to obtain visual error information corresponding to the text error information.

6. The mixed-source multimodal error information detection method according to claim 2, characterized in that: The multiple types of error information include cross-modal consistency error information; The obtaining of the multiple types of error information further includes: The cross-modal consistency error information is obtained based on the inconsistency conversion.

7. The mixed-source multimodal error information detection method according to claim 1, characterized in that: Also includes: The target detection model is improved based on retrieval enhancement and tool learning to obtain an improved target detection model.

8. A mixed-source multimodal error information detection system, characterized in that: include: Information module, used to obtain target information; A detection module, used for inputting the target information into a target detection model to determine whether the target information is error information; The target detection model is a multimodal error information detection model trained based on multiple types of error information, and the multiple types of error information have different information sources.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images