An image forgery detection method based on a multi-modal large language model

Abstract

The present application relates to the technical field of image detection, and provides an image forgery detection method based on a multi-modal large language model, which comprises: inputting a to-be-detected image into the multi-modal large language model; and outputting a true-false identification result of the to-be-detected image by the multi-modal large language model; wherein the multi-modal large language model has three parallel expert branches of physical consistency, semantic consistency and bottom structure clue analysis. The present application perceives semantic information in an image, understands the context logic and physical logic of the image by fully utilizing the powerful prior knowledge, deep semantic understanding ability and chain reasoning ability of the multi-modal large language model, and can perform complex reasoning and explanation on the forged image through natural language.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and more specifically, to an image forgery detection method based on a multimodal large language model. Background Technology

[0002] With the rapid development of generative technologies (such as GANs (Generative Adversarial Networks), diffusion models, autoencoder models, etc.), digital image generation and editing technologies have achieved revolutionary breakthroughs. These technologies can generate visually highly realistic forged images or videos that are almost indistinguishable from real images by the naked eye. While providing powerful and convenient tools, this has also raised significant social concerns. The misuse of this technology has led to risks such as privacy violations, copyright disputes, and the spread of false information, posing a serious threat to personal privacy, the authenticity of news, judicial and social trust, financial security, and even national and social stability.

[0003] As the defender, existing image forgery detection technologies have the following shortcomings:

[0004] (1) Relying on underlying structural clues, it can effectively identify texture anomalies, but it is difficult to handle artifacts that require global reasoning (such as cases that violate the physical laws of light and shadow).

[0005] (2) Poor generalization performance. Since most algorithms are only trained on forged data of the same feature type, the detection performance drops sharply when encountering new forged models with unknown sources or unknown tampering methods.

[0006] (3) Existing artificial intelligence-generated traces involve inconsistencies in multiple aspects, including image content, structure, style and other intrinsic features, showing greater flexibility, diversity and complexity, which current detection methods struggle to cope with;

[0007] (4) Lack of interpretability: The decision-making process of current image forgery detection methods is like a black box, with poor interpretability and difficulty in providing convincing evidence of tampering.

[0008] The emergence of Multi-modal Large Language Models (MLLMs), such as GPT-4V and LLaVA, has demonstrated powerful cross-modal understanding and reasoning capabilities. However, current technologies rarely effectively utilize the deep understanding and chain-like reasoning capabilities of large language models. Furthermore, there is a lack of image forgery detection methods and systems that can simultaneously integrate high-level semantic reasoning information and low-level structural features for forgery detection and analysis, and provide clear, reasonable, and interpretable descriptions. Summary of the Invention

[0009] This invention aims to provide an image forgery detection method based on a multimodal large language model, in order to solve the problems that current image forgery detection methods mostly rely on low-level structural clues (such as noise inconsistency, ghosting effect, JPEG compression artifacts, etc.), and have poor generalization performance and lack of interpretability.

[0010] This invention provides an image forgery detection method based on a multimodal large language model, comprising:

[0011] Input the image to be detected into the multimodal large language model;

[0012] The multimodal large language model outputs the authenticity determination results of the image to be detected;

[0013] The multimodal large language model has three parallel expert branches: physical consistency, semantic consistency, and underlying structural clue analysis.

[0014] In a preferred embodiment, the multimodal large language model outputs the authenticity determination result of the image to be detected, including:

[0015] The image to be detected is input into a shared visual coding module to obtain global and local features;

[0016] The global and local features are input into the physical consistency expert branch, semantic consistency expert branch, and underlying structure clue analysis expert branch in the multi-expert evidence collection module to obtain the physical feature vector, semantic feature vector, and underlying structure feature vector, respectively.

[0017] The physical feature vector, semantic feature vector, and underlying structural feature vector are input into the feature fusion and expert prompt sequence generation module for feature fusion and conversion into an expert prompt sequence.

[0018] The prompt word engineering construction module builds prompt word projects based on expert prompt sequences;

[0019] The large language model decoder module analyzes the prompts provided by the prompt word project to obtain the authenticity determination result of the image to be detected.

[0020] In a preferred embodiment, in the physical consistency expert branch, after the global features are input into the physical consistency expert branch, a lightweight multilayer perceptron outputs a physical feature vector; the activation function in the multilayer perceptron is the ReLU function, and the output head adopts a dual-head output design of regression head and encoding head; the regression head output uses a linear layer to predict physical properties, and the encoding head projects the multilayer perceptron backbone output into a low-dimensional vector, which is used to encode the existence and severity of physical inconsistencies in the entire image.

[0021] In a preferred embodiment, in the semantic consistency expert branch, after the global features and local features are input, they pass through a relation reasoning module based on a fusion attention mechanism. This relation reasoning module first uses a mutual attention layer to treat the local features as... and Using global features as Calculate the enhanced feature vector The enhanced feature vector will then be... The input is fed into a lightweight 2-layer Transformer encoder for inference. Finally, the generated sequence is average-pooled and connected to a linear layer to output a semantic feature vector.

[0022] In a preferred embodiment, the enhanced feature vector is calculated. Represented as:

[0023]

[0024] in, It is a mutual attention layer operation. for function, It is a global feature. It is a local feature. For the transpose of local features, It is the scaling factor.

[0025] In a preferred embodiment, in the low-level structural cue analysis expert branch, after the local features are input into the low-level structural cue analysis expert branch, the 1D sequence features are first readjusted into a 2D feature map, and then the 2D feature map is input into a lightweight convolutional encoder; the convolutional encoder consists of multiple convolutional blocks, each convolutional block including a A convolutional layer is followed by a Batch Normalization (BN) layer, with ReLU as the activation function. Finally, the feature map processed by the convolutional encoder is averaged and then connected to a linear layer to obtain the feature vector of the underlying structure.

[0026] In a preferred embodiment, in the feature fusion and expert prompt sequence generation module, after receiving the physical feature vector, semantic feature vector, and underlying structural feature vector, the feature vector first passes through a feature vector projection layer. This feature vector projection layer processes the feature vectors of the three expert branches through three independent linear layers for word embedding. Then, expert token marking is performed to create expert tokens that the large language model can understand. These tokens are used to indicate the end of the input of physical consistency expert information, semantic consistency expert information, underlying structural clue expert information, and expert information, thereby obtaining the expert prompt sequence.

[0027] In a preferred embodiment, the prompt word engineering construction module constructs a prompt word engineering based on expert prompt sequences, including:

[0028] Define the roles;

[0029] Explain the deeper professional meaning represented by each expert token;

[0030] Introducing a chain of thought forces multimodal large language models to engage in chain-like thinking;

[0031] Provide a template for structured output.

[0032] In a preferred embodiment, the analysis performed by the large language model decoder module includes:

[0033] Based on the prompts, the multimodal large language model decomposes complex reasoning tasks into multiple consecutive intermediate steps before performing comprehensive analysis.

[0034] After comprehensive analysis, autoregression generation is performed to obtain the authenticity identification result of the image to be detected and an identification description report.

[0035] In a preferred embodiment, the multimodal large language model needs to be trained using a training set; the training set includes images, instructions, and expert reports; the images include real images and fake images.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] 1. This invention fully utilizes the powerful prior knowledge, deep semantic understanding capabilities, and chain reasoning capabilities of multimodal large language models to perceive semantic information in images, understand the contextual and physical logic of images, and simultaneously perform complex reasoning and interpretation of forged images through natural language.

[0038] 2. This invention adopts a new paradigm of "collaborative expert consultation" to replace the traditional "black box classifier" approach. Through deep cue word engineering, its capabilities are precisely constrained to the field of image forgery detection. At the same time, it can output a detailed, rigorous, and well-supported identification and description report, which greatly improves the credibility and practicality of forgery detection. Attached Figure Description

[0039] Figure 1 A flowchart of an image forgery detection method based on a multimodal large language model provided in an embodiment of the present invention.

[0040] Figure 2 This is a flowchart of constructing a training set in an embodiment of the present invention.

[0041] Figure 3This is a schematic diagram of constructing the prompt word project in an embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0045] like Figure 1 As shown, this embodiment of the invention provides an image forgery detection method based on a multimodal large language model, including:

[0046] Input the image to be detected into the multimodal large language model;

[0047] The multimodal large language model outputs the authenticity determination results of the image to be detected;

[0048] The multimodal large language model has three parallel expert branches: physical consistency, semantic consistency, and underlying structural clue analysis.

[0049] Therefore, this invention fully utilizes the powerful prior knowledge, deep semantic understanding, and chain reasoning capabilities of large language models to comprehensively determine whether an image is a forgery from three perspectives: physical consistency, semantic logical consistency, and analysis of underlying structural clues. At the same time, it can generate interpretable natural language text.

[0050] The following details the specific implementation of the image forgery detection method based on the multimodal large language model.

[0051] S101, collect datasets, including real image datasets and fake image datasets.

[0052] Real-world image datasets utilize large-scale natural image datasets such as COCO and ImageNet, which provide relatively rich information on natural scenes and physical objects.

[0053] The forged image datasets mainly involve forged image datasets generated by several mainstream generation algorithms, including: general image tampering datasets (such as CASIA), datasets synthesized using diffusion models (Stable Diffusion, DALL-E, etc.) (such as FaceForensics++), forged detection datasets of GANs (data generated using StyleGAN, BigGAN, CycleGAN, etc.) (such as DDL-Datasets), and datasets generated by AIGC (such as DeepAction).

[0054] S102, preprocess the collected dataset.

[0055] The preprocessing includes image normalization and data augmentation.

[0056] The image standardization includes: scaling all images in the dataset to the input size (224*224) adapted to the vit encoder; and converting all images in the dataset to the same format for unified processing.

[0057] The data augmentation includes expanding the dataset through random scaling, adding noise, random blurring, and color adjustments.

[0058] S103 generates instructions and expert reports based on the preprocessed dataset.

[0059] The instructions are generated using a template-based approach. In this embodiment of the invention, a set of predefined templates and rules are used to automatically generate a large number of diverse instructions. The templates that can be used are as follows:

[0060] "Please analyze whether this image has been tampered with."

[0061] "Analyze whether this image is reasonable from the perspectives of lighting, texture, and shadow."

[0062] "Based on the scene information in the image, analyze whether the existence of the objects in this image is reasonable."

[0063] Please explain why you believe this image is fake.

[0064] …

[0065] The expert reports are generated using automated preprocessing and large-model-assisted annotation. For some forged and tampered images, the source dataset provides tampering masks, allowing automated preprocessing to extract forged information and regions. Large-model-assisted annotation first generates a preliminary report using a large model, which is then quickly reviewed and modified by experts to generate a detailed forgery analysis report. This large-model-assisted annotation significantly improves annotation efficiency.

[0066] S104, a training set is built based on the dataset, instructions, and expert reports.

[0067] The images, along with the corresponding instructions and expert reports in the dataset, are serialized into a unified data storage format, such as a JSON file, to obtain the training set, which is used to train a multimodal large language model.

[0068] S200 utilizes a training set to train a multimodal large language model, which has three parallel expert branches: physical consistency, semantic consistency, and low-level structural cue analysis. For example... Figure 1 As shown, it specifically includes:

[0069] S201, input the image into the shared visual coding module to obtain global and local features.

[0070] In this embodiment of the invention, the shared visual coding module is a shared visual coding module based on a VIT-B / 16 encoder. In this shared visual coding module, the input image is first segmented into several image blocks (e.g., a 224*224 input image is segmented into 256 14*14 patch image blocks). Each patch image block is linearly projected to become a token, and a positional encoding is added to the token to obtain a sequence. Simultaneously, a learnable [CLS] token is added to the beginning of the sequence to obtain a token sequence. All token sequences are passed through a series of Transformer encoders to obtain global and local features. The global features correspond to the output feature vector of the [CLS] token (denoted as...). The local features correspond to the feature vectors of the remaining tokens except for the [CLS] token (denoted as...). ), These tokens, corresponding to the number of image patches, preserve the spatial location information and local feature regions of the image.

[0071] S202, input the global features and local features into the physical consistency expert branch, semantic consistency expert branch and underlying structure clue analysis expert branch in the multi-expert evidence collection module to obtain the physical feature vector, semantic feature vector and underlying structure feature vector respectively.

[0072] (1) Physical Consistency Expert Branch.

[0073] The Physical Consistency Expert Branch is responsible for learning and predicting hidden features related to physical properties. Its goal is to analyze whether physical properties such as lighting, shadows, reflections, and perspective in an image scene are consistent.

[0074] Specifically, the global features After inputting the physical consistency expert branch, the data passes through a lightweight multilayer perceptron (MLP) and outputs a multidimensional vector, namely the physical feature vector. The multilayer perceptron is used to estimate multiple physical properties of the main light source. The activation function in the multilayer perceptron is the ReLU function, and the output head adopts a dual-head output design of regression head and encoding head. The regression head output uses a linear layer to predict physical properties. The encoding head projects the multilayer perceptron backbone output into a low-dimensional vector, which is used to encode the existence and severity of physical inconsistencies in the entire image. It is a highly abstract feature that will be used later for large language models to learn physical language interpretation.

[0075] (2) Semantic Consistency Expert Branch.

[0076] The semantic consistency expert branch is responsible for analyzing the semantic relationships between objects, focusing on detecting semantic errors in images that violate common sense, logic, and world knowledge. Examples include the co-occurrence of objects that contradict common sense, unreasonable size ratios, and incorrect scene context. The goal of this semantic consistency expert branch is to analyze the rationality of the semantic relationships between objects, people, and the environment in an image scene.

[0077] Specifically, global features and local features After the semantic consistency expert branch is input, it passes through a relational reasoning module based on a fusion attention mechanism. This relational reasoning module first uses a mutual attention layer to process local features. As and global features As Calculate the enhanced feature vector The formula is shown below. This operation allows the global scene to query the local region to obtain information on whether the local region matches the global state.

[0078]

[0079] in, It is a mutual attention layer operation. for function, For the transpose of local features, It is the scaling factor.

[0080] Then the output of the mutual attention layer, i.e. the enhanced feature vector, is... The input is fed into a lightweight 2-layer Transformer encoder for further inference. Finally, the generated sequence is average-pooled and connected to a linear layer to output a semantic feature vector. This semantic feature vector serves as the semantic logic language of the large language model for subsequent input.

[0081] (3) Expert branch of underlying structure clue analysis.

[0082] The underlying structural cue analysis expert branch focuses on detecting pixel-level and texture-level statistical anomalies in images. The goal of this underlying structural cue analysis expert branch is to analyze whether there are inconsistencies in features such as noise, texture, artifacts, and edge sharpness between different regions of the image.

[0083] Specifically, the local features After inputting the low-level structural clues analysis expert branch, the 1D sequence features are first readjusted into a 2D feature map. The adjusted 2D feature map is then input into a lightweight convolutional encoder, which leverages the ability and advantages of convolutional neural networks (CNNs) in capturing local texture features of images. In this embodiment of the invention, the convolutional encoder consists of multiple convolutional blocks, each including a... A convolutional layer is followed by a batch normalization (BN) layer, with ReLU chosen as the activation function. Finally, the feature map processed by the convolutional encoder is average-pooled and then connected to a linear layer to obtain the feature vector of the underlying structure. This feature vector is the texture language of the large language model.

[0084] S203, input the physical feature vector, semantic feature vector and underlying structural feature vector into the feature fusion and expert prompt sequence generation module to perform feature fusion and convert it into an expert prompt sequence;

[0085] The feature fusion and expert prompt sequence generation module integrates feature vectors from different expert domains into an expert prompt sequence that the large model can understand.

[0086] Specifically, the feature fusion and expert suggestion sequence generation module receives physical feature vectors. semantic feature vector and underlying structural feature vectors Afterwards, the input first passes through a feature vector projection layer, which projects the feature vectors from the feature space to the word embedding space. This layer uses three independent linear layers to process the feature vectors of the three expert branches, performing word embedding. This feature vector projection layer maps the expert feature vectors from the original high-dimensional feature space to the word embedding space of the large language model. Next, expert tokens are used to create expert tokens that the large language model can understand, thus obtaining the expert prompt sequence. In this embodiment, four expert tokens are defined in the vocabulary of the large language model: [PHYS], [SEM], [LOW], and [ / EXP]. [PHYS], [SEM], and [LOW] are start markers, indicating that the following information is physical consistency expert information, semantic consistency expert information, and low-level structural clue expert information, respectively. [ / EXP] serves as the end marker, defining the end of the expert information input.

[0087] S204, The prompt word project construction module constructs a prompt word project Prompt based on the expert prompt sequence.

[0088] The prompt word engineering construction module is the operational procedure of the large language model. It guides the subsequent reasoning process of the large language model through the design of system prompts, user commands, and output formats. For example... Figure 3 As shown, the process begins with defining roles, followed by explaining the deeper professional meaning represented by each expert token. Then, the Chain of Thought (CoT) is introduced to force the multimodal large language model to engage in chain-like thinking. Finally, a structured output template is provided, allowing users to input commands for analysis.

[0089] S205, the large language model decoder module, is the decision-making brain of the entire image forgery detection process. As mentioned above, the entire forgery detection process is similar to a sophisticated "simulated expert consultation" process: First, the multi-expert evidence collection module (physical, semantic, and underlying structure) acts as a professional "detection instrument," extracting high-dimensional, abstract feature vectors (such as physical feature vectors) from the image. semantic feature vector and underlying structural feature vectors These vectors encode preliminary evidence of whether anomalies exist in their respective dimensions. Subsequently, the feature projection layer plays a crucial "translator" role, using trainable linear transformations to "translate" these abstract feature vectors into "expert pseudo-tokens" that the large language model can understand. These pseudo-tokens are essentially a piece of internal language embedded in the semantic space of the large model, carrying the discoveries of experts. Next, the prompt word engineering module, acting as an "expert assistant," dynamically embeds these pseudo-tokens as core evidence into a pre-defined, structured prompt word template. Then, the large language model decoder module, acting as the "chief expert," guides the multimodal large language model, based on the prompt words provided by the prompt word engineering prompt, to decompose the complex reasoning task into multiple consecutive intermediate steps before comprehensive analysis. Specifically, when it needs to write the "physical dimension" section, its attention is highly focused on the [PHYS] token and subsequent pseudo-tokens in the input sequence, interpreting physical anomalies and converting them into natural language descriptions. This process is repeated for all expert evidence, ultimately following the instruction format. After the comprehensive analysis is completed, autoregressive generation is performed to obtain the authenticity determination result of the image to be detected, as well as a rigorous and well-supported identification description report.

[0090] The specific training methods for training a multimodal large language model using the training set can adopt common training methods for large language models, which will not be elaborated here.

[0091] S300, by training a multimodal large language model using a training set, can use the trained multimodal large language model to determine the authenticity of the image to be detected. The image to be detected also needs to undergo image standardization preprocessing, similar to the images in the training set, to enable the multimodal large language model to quickly determine the authenticity of the image.

[0092] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the image forgery detection method based on a multimodal large language model provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 4 As shown, the electronic device may include:

[0093] At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 4 The example used is the connection between the processor and memory via a bus. The bus... Figure 4 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 4The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0094] In this embodiment of the invention, the memory stores instructions that can be executed by at least one processor. By executing the instructions stored in the memory, at least one processor can execute the image forgery detection method based on a multimodal large language model discussed above.

[0095] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0096] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0097] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the image forgery detection method based on a multimodal large language model disclosed in the embodiments of this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0098] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia cards, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), and electrically erasable programmable read-only memory (EPROM). Only memory (EEPROM), magnetic storage, magnetic disks, optical disks, etc. A memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. The memory in embodiments of this invention can also be a circuit or any other device capable of performing storage functions for storing program instructions and / or data.

[0099] By designing and programming the processor, the code corresponding to the image forgery detection method based on a multimodal large language model described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the method described in the foregoing embodiments during runtime. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0100] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform an image forgery detection method based on a multimodal large language model as described above.

[0101] In some alternative embodiments, the present invention also provides that various aspects of the image forgery detection method based on a multimodal large language model can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the image forgery detection method based on a multimodal large language model according to various exemplary embodiments of the present invention as described above.

[0102] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0103] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can be implemented in one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs) containing computer-usable program code. The form of a computer program product implemented on ROM, optical memory, etc.

[0104] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0106] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-modal large language model-based image forgery detection method, characterized in that, The method comprises the following steps: inputting an image to be detected into a multimodal large language model; the multimodal large language model outputs a true or false identification result of the image to be detected; wherein the multimodal large language model has three parallel expert branches of physical consistency, semantic consistency and bottom structure clue analysis; the multimodal large language model outputs a true or false identification result of the image to be detected, which comprises: inputting the image to be detected into a shared visual coding module to obtain global features and local features; inputting the global features and the local features into the physical consistency expert branch, the semantic consistency expert branch and the bottom structure clue analysis expert branch in the multi-expert forensics module to obtain physical feature vectors, semantic feature vectors and bottom structure feature vectors respectively; inputting the physical feature vectors, the semantic feature vectors and the bottom structure feature vectors into a feature fusion and expert prompt sequence generation module for feature fusion and conversion into an expert prompt sequence; an expert prompt sequence construction module constructs an expert prompt sequence based on the expert prompt sequence; a large language model decoder module analyzes the prompt words provided by the expert prompt sequence construction module to obtain a true or false identification result of the image to be detected; in the physical consistency expert branch, after inputting the global features into the physical consistency expert branch, a lightweight multilayer perceptron outputs the physical feature vectors; in the multilayer perceptron, the activation function is ReLu function, and the output head adopts a double-head output design of regression head and coding head; the regression head output adopts a linear layer to predict physical properties, and the coding head is to project the output of the multilayer perceptron trunk into a low-dimensional vector, which is used to encode the existence and severity of physical inconsistency in the entire image; In the semantic consistency expert branch, after the global feature and the local feature are input into the semantic consistency expert branch, a relationship reasoning module based on a fusion attention mechanism is used. The relationship reasoning module first uses a mutual attention layer to take the local feature as and takes the global feature as to calculate an enhanced feature vector ; then the enhanced feature vector is input into a 2-layer lightweight Transformer encoder for reasoning, and finally, the sequence generated after reasoning is subjected to average pooling and a linear layer to output a semantic feature vector; In the bottom structure clue analysis expert branch, after inputting the local feature into the bottom structure clue analysis expert branch, first, the 1D sequence feature is re-adjusted into a 2D feature map, and then the 2D feature map is input into a lightweight convolutional encoder; the convolutional encoder is composed of multiple convolutional blocks, each convolutional block includes a convolutional layer followed by a BN normalization layer, and the activation function is selected as a ReLu function; finally, the feature map processed by the convolutional encoder is subjected to average pooling followed by a linear layer to obtain a bottom structure feature vector.

2. The multi-modal large language model-based image forgery detection method of claim 1, wherein, computing an enhanced feature vector is represented as: wherein, is a mutual attention layer operation, is a function, is a global feature, is a local feature, is a transpose of the local feature, is a scaling factor.

3. The multimodal large language model-based image forgery detection method of claim 1, wherein, in the feature fusion and expert prompt sequence generation module, after receiving the physical feature vectors, the semantic feature vectors and the bottom structure feature vectors, first pass through a feature vector projection layer, which sets up three independent linear layers to process the feature vectors of the three expert branches for word embedding; then perform expert token marking to create expert tokens that can be understood by the large language model, which are used to indicate physical consistency expert information, semantic consistency expert information, bottom structure clue expert information and the end of expert information input, thereby obtaining an expert prompt sequence.

4. The multimodal large language model-based image forgery detection method according to claim 3, characterized in that, the expert prompt sequence construction module constructs a prompt word engineering based on the expert prompt sequence, which comprises: define roles; explain the deep professional meaning represented by each expert token; introduce a thinking chain to force the multimodal large language model to perform chain thinking; provide a template for structured output. 5.The multimodal large language model based image forgery detection method of claim 1, wherein, the analysis performed by the large language model decoder module comprises: decompose the complex reasoning task into multiple consecutive intermediate steps and then perform comprehensive analysis according to the prompt words to guide the multimodal large language model; after the comprehensive analysis is completed, perform autoregressive generation to obtain a true or false identification result of the image to be detected and an identification description report.

6. The multimodal large language model-based image forgery detection method according to any one of claims 1-5, characterized in that, the multimodal large language model needs to be trained using a training set; the training set is a training set comprising images, instructions and expert reports; the images include real images and fake images.

Images