A multi-modal fusion counterfeit image identification method, device and medium

Abstract

The present application relates to the field of forged image recognition, and provides a multi-modal fusion forged image recognition method, device and medium, which comprises the following steps: extracting multi-modal features containing vision, physics and semantics; adaptively fusing the multi-modal features to obtain multi-modal fusion features; performing bidirectional trajectory verification based on the multi-modal fusion features to obtain bidirectional trajectory differences; recognizing forged images based on the bidirectional trajectory differences; tracing the forged areas in the forged images by using the bidirectional trajectory differences, and outputting the tracing evidence. The present application can solve the problems of low recognition accuracy, poor generalization to unknown forgery, and difficult interpretation of recognition results, etc.

Description

Technical Field

[0001] This invention relates to the field of forged image recognition, and more specifically, to a multimodal fusion method, device, and medium for forged image recognition. Background Technology

[0002] Currently, image forgery technology is developing rapidly, from early simple pixel manipulation to high-fidelity face generation and deepfake face swapping based on GAN (Generative Adversarial Network), and then to full-scene image forgery combined with Diffusion model. The visual difference between forged images and real images is becoming smaller and smaller, posing a huge challenge to the identification of information authenticity.

[0003] Existing forged image recognition technologies have significant shortcomings: First, feature extraction is limited to a single dimension. Most methods rely solely on pixel statistical features or shallow texture features, making it difficult to capture anomalies in deeper dimensions such as semantic logic and physical attributes of forged images. For example, they cannot identify implicit forgery traces such as "contradictions between the lighting direction of the face and the background" or "mismatch between the position of the object's shadow and the light source." Second, feature fusion methods are simplistic. Some multi-feature fusion methods only use splicing or weighted summation fusion modes, failing to consider the complementarity and correlation of different modal features. This results in high feature redundancy after fusion, dilution of effective information, and difficulty in improving recognition accuracy. Third, generalization ability is weak. Models trained for specific forgery methods suffer a significant drop in recognition accuracy when facing new forgery techniques because they do not cover the corresponding modal features. For instance, models trained for GAN-forged images often have an accuracy rate of less than 65% for Diffusion-forged images. Fourth, the interpretability is poor. Most existing recognition models are "black box" models (such as deep neural networks), which only output binary classification results of "fake / real". They cannot locate the fake area, explain the fake type (such as face swapping, splicing) and judge the fake technology principle, making the recognition results difficult to be accepted by the judicial, news and other fields.

[0004] Therefore, there is an urgent need for an image forgery recognition technology that can balance high accuracy, strong generalization and high interpretability to solve the above-mentioned technical pain points. Summary of the Invention

[0005] The present invention aims to provide a multimodal fusion method, device and medium for forged image recognition, in order to solve the problems of low recognition accuracy, poor generalization to unknown forgeries and difficulty in interpreting recognition results in current forged image detection.

[0006] In a first aspect, the present invention provides a multimodal fusion method for identifying forged images, comprising:

[0007] Extract multimodal features that include visual, physical, and semantic aspects;

[0008] Adaptive fusion of multimodal features yields multimodal fused features;

[0009] Bidirectional trajectory verification is performed based on multimodal fusion features to obtain bidirectional trajectory differences;

[0010] Identifying forged images based on bidirectional trajectory differences;

[0011] By utilizing bidirectional trajectory differences, the forged regions in the forged image are traced back to their source, and the traceability evidence is output.

[0012] In a preferred embodiment, the extraction of multimodal features including visual, physical, and semantic features includes:

[0013] Visual modal features are extracted, and the sub-features of the visual modal features include pixel-level features and texture-level features;

[0014] Physical modal features are extracted, and the sub-features of the physical modal features include illumination features and shadow features;

[0015] Extract semantic modality features.

[0016] In a preferred embodiment, the adaptive fusion of multimodal features to obtain multimodal fused features includes:

[0017] Construct a modality weight prediction subnetwork, input the information entropy of each multimodal feature into the modality weight prediction subnetwork, and obtain the weight coefficient of each multimodal feature;

[0018] A cross-modal attention mechanism is adopted to calculate the correlation matrix of different sub-features among multimodal features, and the corresponding weight coefficients are adjusted based on the correlation matrix through the attention mechanism.

[0019] Multimodal fusion features are obtained by fusing various multimodal features based on weight coefficients.

[0020] In a preferred embodiment, the step of performing bidirectional trajectory verification based on multimodal fusion features to obtain bidirectional trajectory differences includes:

[0021] Using multimodal fusion features as the initial state of the positive trajectory, we perform noise-based positive trajectory calculations at several time steps to simulate the natural degradation process of a real image towards noise.

[0022] Using random noise as the initial state of the reverse trajectory, the reverse trajectory is calculated based on noise at several time steps to simulate the generation process from noise to image features.

[0023] The difference between the forward and reverse trajectories at each time step is calculated using the Euclidean norm, and the total difference index of the two-way trajectory is obtained by accumulating the difference.

[0024] In a preferred embodiment, the step of performing a forward trajectory calculation based on noise over several time steps is expressed as follows:

[0025]

[0026] in, For the first positive trajectory t The positive state of the time step. For the first positive trajectory t The positive state at time step -1. For the first t The dynamic noise figure of the time step. Gaussian noise that conforms to a standard normal distribution is used to calculate the forward trajectory.

[0027] In a preferred embodiment, the step of performing noise-based reverse trajectory calculation over several time steps is expressed as follows:

[0028]

[0029] in, For the reverse trajectory, the first t The reverse state of the time step, For the reverse trajectory, the first t The reverse state at time step +1 For the first t The dynamic noise figure of the time step. , For the first t Adaptive sampling variance at time steps For sampling noise, For the first t Prediction noise at time steps.

[0030] In a preferred embodiment, the method for identifying forged images based on bidirectional trajectory differences includes:

[0031] The total difference index of the two-way trajectory is compared with the optimal judgment threshold. If the total difference index of the two-way trajectory exceeds the optimal judgment threshold, the image is judged to be a fake image; otherwise, it is a real image.

[0032] The optimal judgment threshold is obtained by training with real image and fake image samples.

[0033] In a preferred embodiment, the step of tracing the forged region in the forged image using bidirectional trajectory differences and outputting tracing evidence includes:

[0034] The time steps in which the bidirectional trajectory difference exceeds a local threshold are mapped back to image blocks in the image and marked as suspected forgery blocks. Using a region growing algorithm, adjacent image blocks with a difference higher than a first set value are merged to obtain the forgery region, and the forgery region is marked on the image.

[0035] Calculate the similarity between the multimodal fusion features of the forged region and the modal features of each category in the multimodal forgery type feature library. The category with the highest similarity exceeding the second set value is the forgery type determination result.

[0036] Output source tracing evidence, including a modal difference heatmap, a feature comparison table, and a judgment report; the modal difference heatmap is used to visually display the difference distribution under each modality; the feature comparison table is used to quantify the differences in modal features between the forged region and the real region; the judgment report includes the total difference of the two-way trajectory, the forgery type judgment result, and the confidence level calculated based on similarity.

[0037] In a second aspect, the present invention provides an electronic device, comprising:

[0038] At least one processor; and a memory communicatively connected to said at least one processor;

[0039] The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the instructions stored in the memory to perform the above-described method.

[0040] Thirdly, the present invention provides a computer-readable storage medium for storing instructions that, when executed, enable the above-described method to be implemented.

[0041] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0042] 1. High recognition accuracy: The multimodal feature system covers visual, physical, and semantic dimensions. The adaptive fusion mechanism enhances effective features. Combined with bidirectional trajectory verification, it can improve the recognition accuracy of known forgery methods (GAN generation, Deepfake face swapping) and complex forged images such as low-resolution (256×256) and multi-region tampering.

[0043] 2. Strong generalization ability: It does not rely on the exclusive features of specific forgery methods, but captures common anomalies of forged images through multimodal fusion (such as contradictory physical properties and confusing semantic logic). Combined with bidirectional trajectory difference quantization, it can improve the recognition accuracy of new forgery technologies (such as Diffusion full-scene forgery), reduce generalization error, and solve the problem of "model failure caused by the iteration of forgery technology".

[0044] 3. Excellent interpretability: It achieves pixel-level forgery area localization based on multimodal feature differences, completes type determination by combining modal feature library, and outputs a three-dimensional evidence chain that meets the needs of the judicial and news fields for "traceability and verifiability", which can improve the confidence of the judgment results.

[0045] 4. Good engineering practicality: It adopts a lightweight network (MobileNetV4, SegNeXt) and an adaptive sampling step size (T=50 for simple images, T=80 for complex images), supports batch processing (processing 3-5 images per second), and can be deployed on cloud servers and edge devices (such as mobile devices). Attached Figure Description

[0046] Figure 1 This is a flowchart of a multimodal fusion-based method for identifying forged images, provided as an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0050] Example

[0051] like Figure 1 As shown, this embodiment of the invention provides a multimodal fusion method for identifying forged images, including:

[0052] S100 extracts multimodal features that include visual, physical, and semantic elements;

[0053] S200, adaptively fuses multimodal features to obtain multimodal fused features;

[0054] S300, based on multimodal fusion features, performs bidirectional trajectory verification to obtain bidirectional trajectory differences;

[0055] S400, based on bidirectional trajectory difference identification of forged images;

[0056] The S500 uses bidirectional trajectory differences to trace the forged regions in a forged image and outputs evidence of the forgery.

[0057] The following details the specific implementation of the multimodal fusion-based forgery image recognition method.

[0058] S100 extracts multimodal features including visual, physical, and semantic elements:

[0059] For the input image to be identified, a three-dimensional modal feature system of "visual-physical-semantic" is constructed to extract multi-dimensional and highly complementary features, comprehensively uncovering suspicious elements in the image and improving the recognition accuracy of various complex forgeries. Specifically, this includes:

[0060] Visual modal features are extracted, and the sub-features of these visual modal features include pixel-level features and texture-level features. Specifically, the pixel-level features are calculated by taking the mean, variance, skewness, and kurtosis of image blocks (e.g., 16×16 pixel blocks), as well as the correlation coefficients of the RGB three channels, resulting in a 64-dimensional pixel feature vector. The texture-level features are obtained by using a pre-trained lightweight convolutional network, MobileNetV4, to extract deep texture information from the image, resulting in a 256-dimensional texture feature vector. The focus is on capturing features such as "repeated textures and blurred edges" in forged images.

[0061] Physical modal features are extracted, focusing on the consistency of physical attributes in the image. The sub-features of these physical modal features include illumination features and shadow features. The illumination features are calculated using an illumination estimation model (selecting an appropriate model as needed, such as a CNN-based regression model) to determine the illumination direction, intensity, and color temperature of each region of the image, resulting in a 48-dimensional illumination feature vector. The shadow features are extracted as a 32-dimensional shadow feature vector by analyzing the positional relationship between the object's outline and the shadow, as well as the grayscale gradient. It is used to identify forgery traces such as "shadows without light source" and "shadow shapes that do not match objects".

[0062] Extracting Semantic Modal Features: Mining Anomalies from a Semantic Logic Level. A pre-trained semantic segmentation model, SegNeXt, is used to semantically annotate images (e.g., categories such as faces, bodies, buildings, and natural landscapes), generating 128-dimensional semantic category features. Simultaneously, the logical correlation between semantic regions is calculated (e.g., "connection integrity between face and neck" and "occlusion logic between object and background"), forming 64-dimensional semantic association features. The semantic modal features obtained by merging the semantic category features and semantic association features are a 192-dimensional semantic feature vector. .

[0063] S200, adaptive fusion of multimodal features to obtain multimodal fused features:

[0064] A two-layer fusion mechanism of "modal weight learning + cross-modal attention" is designed to address the issue of varying importance of different modal features in different forgery scenarios. The specific process is as follows:

[0065] Modality weight learning: Construct a modality weight prediction subnetwork, input the information entropy of each multimodal feature into the modality weight prediction subnetwork (the higher the information entropy, the richer the effective discriminative information contained in the feature), and output the weight coefficients of the visual modality features. Weighting coefficients of physical modal characteristics Weight coefficients of semantic modal features ,satisfy + + =1. For example, in Deepfake face-swapping scenarios, the weight coefficient of physical modal features (lighting, shadows) is 1. The weights of visual modal features can be increased to 0.45. The weight of the semantic modality feature is 0.35. The weight coefficient is 0.2; in image stitching scenarios, the weight coefficient of semantic modality features is... It can be improved to 0.4, with the weight coefficients of visual modality features and physical modality features being 0.35 and 0.25, respectively.

[0066] Cross-modal attention fusion: For sub-features under each multimodal feature (such as pixel-level features and texture-level features under visual modality), a cross-modal attention mechanism is adopted to calculate the correlation matrix between different sub-features of the multimodal feature, strengthen the sub-features with strong discriminative power, and adjust the corresponding weight coefficients based on the correlation matrix through the attention mechanism. For example, when there is a strong correlation between the illumination feature under physical modality and the edge gray-level gradient in the texture-level feature under visual modality (such as abnormal edge gradient caused by abnormal illumination), the corresponding weight coefficients are adjusted through the attention mechanism.

[0067] The multimodal features are fused based on weight coefficients, i.e., the multimodal features are multiplied by their corresponding weight coefficients and then summed to obtain a 512-dimensional multimodal fusion feature. .

[0068] S300, based on multimodal fusion features, performs bidirectional trajectory verification to obtain bidirectional trajectory differences:

[0069] Multimodal fusion features With this as its core, a bidirectional trajectory of forward degradation and reverse generation is constructed. The degree of image forgery is quantified by the difference between the two trajectories. Combined with reasonable feature analysis and knowledge reasoning mechanisms, anomalies are discovered from the perspective of more fundamental changes in image features. Even when faced with unknown forgery methods, the differences between these multimodal features and normal image features can be used to initially determine whether an image is suspected of being forged. Furthermore, it continuously learns and adapts to new forgery situations, enhancing its ability to cope with unknown forgery methods.

[0070] Forward trajectory construction (realistic image degradation simulation): fusing multimodal features As the initial state of the positive trajectory The condition variable c is set to "physical-semantic consistency constraints of the real image" (e.g., consistent lighting and shadow directions, normal logical association of semantic regions). The total sampling time step is set to T=80 as needed, and a noise-based forward trajectory is calculated for T time steps to simulate the natural degradation process of the real image towards noise. The noise-based forward trajectory calculation is expressed as follows:

[0071]

[0072] in, For the first positive trajectory t ( t The positive states of the steps = 1, 2, ..., T) For the first positive trajectory t -1 step positive state, For the first t The dynamic noise figure of the step (linearly decreased from 0.98 to 0.02). Gaussian noise that conforms to a standard normal distribution is used to calculate the forward trajectory.

[0073] Reverse trajectory construction (simulation of fake image generation): with random noise The initial state of the reverse trajectory The condition variable c is consistent with the forward trajectory. A noise-based reverse trajectory is calculated at time step T to simulate the generation process from noise to image features. The noise-based reverse trajectory calculation is expressed as:

[0074]

[0075] in, For the reverse trajectory, the first t ( t The reverse state of time steps = 1, 2, ..., T) For the reverse trajectory, the first t The reverse state at time step +1 For the first t The dynamic noise figure of the step. , For the first t Adaptive sampling variance at time steps For sampling noise, For the first t The prediction noise at each time step is predicted using the Attention U-Net model.

[0076] Trajectory difference calculation: The difference between the forward and reverse trajectories at each time step is calculated using the Euclidean norm, and the total difference index of the two-way trajectory is obtained by accumulating the differences. D , is represented as:

[0077]

[0078] in, This represents the Euclidean norm.

[0079] S400, based on bidirectional trajectory difference identification of forged images:

[0080] The total difference index of the two-way trajectory is compared with the optimal judgment threshold. If the total difference index of the two-way trajectory exceeds the optimal judgment threshold (i.e. If the image is fake, then it is determined to be a fake image; otherwise (i.e.) (This is a real image.)

[0081] S500 uses bidirectional trajectory differences to trace the source of forged regions in a forged image and outputs evidence of the source:

[0082] By generating visually interpretable reports, using charts and graphs to display the contribution of different modal features in the recognition process, the relationships between features, and the correlation path with forgery detection, users can easily understand and trust the recognition results. Specifically:

[0083] Falsifying regional positioning: Distinguishing between two-way trajectories Exceeding the local threshold (Set according to needs and actual application, such as) The time step is mapped back to image blocks in the image and marked as suspected forgery blocks. Using a region growing algorithm, adjacent image blocks with a difference higher than a first preset value (set according to needs and practical applications, such as 0.6) are merged using the suspected forgery blocks as seeds to obtain the complete forgery region. The forgery region is then marked on the image with a positioning accuracy of 8×8 pixels. The forgery region can be marked with a red bounding box, and the intensity of the red can correspond to the magnitude of the difference between the two-way trajectories; the darker the red, the greater the difference. This red is just an example; other colors can be used as needed.

[0084] Forgery type determination: The similarity between the multimodal fusion features of the forged region and the modal features of each category in the multimodal forgery type feature library is calculated. The category with the highest similarity exceeding a second preset value (set according to needs and actual application, such as 0.82) is the forgery type determination result. The multimodal forgery type feature library is pre-constructed and contains modal features of each category of forgery type, such as typical modal features of four forgery types: GAN generation, Deepfake face swapping, image stitching, and content tampering (e.g., "high texture repetition (visual modality)" for GAN-generated images, "abnormal lighting direction (physical modality)" for Deepfake face swapping, and "discontinuous semantic region boundaries (semantic modality)" for image stitching).

[0085] Source tracing evidence output: The output source tracing evidence includes modal difference heatmaps, feature comparison tables, and judgment reports;

[0086] The modal difference heatmap is used to visually display the difference distribution under each mode. Similar to the bounding box annotation of the aforementioned fake area, the darker the red, the greater the difference.

[0087] The feature comparison table is used to quantify the differences in modal features between the fake region and the real region; for example, "the lighting direction of the fake region is 45°, the lighting direction of the real background is 135°, and the deviation is 90°."

[0088] The determination report includes the total difference between the two-way trajectories. D The results of forgery type determination and confidence level calculated based on similarity.

[0089] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the multimodal fusion forged image recognition method flow provided in the above embodiments of the present invention. In one embodiment, the electronic device may be a server, a terminal device, or other electronic device. Figure 2 As shown, the electronic device may include:

[0090] At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 2 The example used is the connection between the processor and memory via a bus. The bus... Figure 2 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 2 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0091] In this embodiment of the invention, the memory stores instructions that can be executed by at least one processor. By executing the instructions stored in the memory, at least one processor can execute a multimodal fusion forgery image recognition method described above.

[0092] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0093] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0094] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the multimodal fusion forged image recognition method disclosed in the embodiments of this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0095] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0096] By designing and programming the processor, the code corresponding to the multimodal fusion forged image recognition method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the method described in the foregoing embodiments during runtime. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0097] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a multimodal fusion method for identifying forged images as described above.

[0098] In some alternative embodiments, the present invention also provides a multimodal fusion method for identifying forged images that can also be implemented as a program product comprising program code that, when the program product is run on a device, causes the control device to perform the steps of the multimodal fusion method for identifying forged images according to various exemplary embodiments of the present invention as described above.

[0099] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0100] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal fusion method for identifying forged images, characterized in that, include: Extract multimodal features that include visual, physical, and semantic aspects; Adaptive fusion of multimodal features yields multimodal fused features; Bidirectional trajectory verification is performed based on multimodal fusion features to obtain bidirectional trajectory differences; Identifying forged images based on bidirectional trajectory differences; By utilizing bidirectional trajectory differences, the forged regions in the forged images are traced, and the evidence for tracing the source is output. The extraction includes multimodal features encompassing visual, physical, and semantic aspects, including: Visual modal features are extracted, and the sub-features of the visual modal features include pixel-level features and texture-level features; Physical modal features are extracted, and the sub-features of the physical modal features include illumination features and shadow features; Extract semantic modality features; The adaptive fusion of multimodal features yields multimodal fusion features, including: Construct a modality weight prediction subnetwork, input the information entropy of each multimodal feature into the modality weight prediction subnetwork, and obtain the weight coefficient of each multimodal feature; A cross-modal attention mechanism is adopted to calculate the correlation matrix of different sub-features among multimodal features, and the corresponding weight coefficients are adjusted based on the correlation matrix through the attention mechanism. Multimodal fusion features are obtained by fusing various multimodal features based on weight coefficients; The bidirectional trajectory verification based on multimodal fusion features, to obtain the bidirectional trajectory difference, includes: Using multimodal fusion features as the initial state of the positive trajectory, we perform noise-based positive trajectory calculations at several time steps to simulate the natural degradation process of a real image towards noise. Using random noise as the initial state of the reverse trajectory, the reverse trajectory is calculated based on noise at several time steps to simulate the generation process from noise to image features. The difference between the forward and reverse trajectories at each time step is calculated using the Euclidean norm, and the total difference index of the two-way trajectory is obtained by accumulating the results. The method of tracing the forged region in a forged image using bidirectional trajectory differences and outputting tracing evidence includes: The time steps in which the bidirectional trajectory difference exceeds a local threshold are mapped back to image blocks in the image and marked as suspected forgery blocks. Using a region growing algorithm, adjacent image blocks with a difference higher than a first set value are merged to obtain the forgery region, and the forgery region is marked on the image. Calculate the similarity between the multimodal fusion features of the forged region and the modal features of each category in the multimodal forgery type feature library. The category with the highest similarity exceeding the second set value is the forgery type determination result. Output source tracing evidence, including a modal difference heatmap, a feature comparison table, and a judgment report; the modal difference heatmap is used to visually display the difference distribution under each modality; the feature comparison table is used to quantify the differences in modal features between the forged region and the real region; the judgment report includes the total difference of the two-way trajectory, the forgery type judgment result, and the confidence level calculated based on similarity.

2. The multimodal fusion method for identifying forged images according to claim 1, characterized in that, The calculation of the forward trajectory based on noise at several time steps is expressed as follows: in, For the first positive trajectory t The positive state of the time step. For the first positive trajectory t The positive state at time step -1. For the first t The dynamic noise figure of the time step. Gaussian noise that conforms to a standard normal distribution is used to calculate the forward trajectory.

3. The multimodal fusion method for identifying forged images according to claim 1, characterized in that, The calculation of the reverse trajectory based on noise at several time steps is expressed as follows: in, For the reverse trajectory, the first t The reverse state of the time step, For the reverse trajectory, the first t The reverse state at time step +1 For the first t The dynamic noise figure of the time step. , For the first t Adaptive sampling variance at time steps For sampling noise, For the first t Prediction noise at time steps.

4. The multimodal fusion method for identifying forged images according to claim 1, characterized in that, The method for identifying forged images based on bidirectional trajectory differences includes: The total difference index of the two-way trajectory is compared with the optimal judgment threshold. If the total difference index of the two-way trajectory exceeds the optimal judgment threshold, the image is judged to be a fake image; otherwise, it is a real image. The optimal judgment threshold is obtained by training with real image and fake image samples.

5. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which executes the instructions stored in the memory to perform the method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store instructions that, when executed, cause the method as described in any one of claims 1-4 to be implemented.

Images