A method and system for identifying fake images based on multimodal analysis

Abstract

The application discloses a false image identification method and system based on multi-modal analysis, belonging to the technical field of deep learning, comprising: acquiring a preprocessed image; constructing a deep neural network of a false image based on a multi-modal image generation technology, the deep neural network comprising: a backbone network module, a multi-modal feature extraction module, a self-adaptive attention module, a dynamic fusion module, a classification module, and a visualization module; obtaining a true-false image identification model according to the deep neural network; and processing the preprocessed image according to the true-false image identification model to obtain an image classification visualization result. The application introduces a context-aware dynamic feature weight distribution mechanism, which can intelligently adjust the importance of different modalities according to the image content characteristics, and simultaneously adopts a hierarchical multi-scale feature integration framework to effectively capture different resolution levels of counterfeit traces. The accuracy, efficiency and interpretability of image identification are improved.

Description

Technical Field

[0001] This application belongs to the field of deep learning technology, specifically relating to a method and system for identifying fake images based on multimodal analysis. Background Technology

[0002] With the rapid development of deep learning models such as generative adversarial networks (GANs), image generation technology has made significant progress. These technologies can generate highly realistic and lifelike images by simulating features such as texture, lighting, and structure in the real world, thus finding widespread application in entertainment, advertising, social media, and other fields, bringing unprecedented impact. However, as generation technology matures, identifying fake images has become increasingly difficult, especially in high-resolution, highly detailed images, making manual judgment of their authenticity almost impossible. The proliferation of fake images may pose a serious threat to social security, particularly in areas such as law, news reporting, and digital identity authentication, where their use could lead to trust crises, legal disputes, and disruption of social order.

[0003] While current fake image detection technologies have made some progress, they still have some significant shortcomings. Specifically, traditional image detection methods mainly suffer from the following deficiencies:

[0004] (1) Insufficient accuracy: Most existing identification methods rely on single modal features, such as analysis based on pixel-level texture features or frequency domain features. Although these methods can achieve good results in some simple scenarios, they cannot fully capture the multidimensional information of images, such as spatial structure, deep semantics, and fine-grained texture changes. Especially in complex scenarios, multi-source image synthesis, and image enhancement applications, traditional methods often encounter performance bottlenecks and cannot effectively deal with various variants of fake images. Therefore, how to design a diversified model that can integrate multiple modal features to improve the accuracy of image identification remains an urgent challenge.

[0005] (2) Inefficiency: With the increase in image resolution and the surge in data volume, traditional image identification methods often face high computational complexity when processing high-resolution images. For example, traditional methods based on pixel-level analysis require a large number of local feature calculations and matching of images, resulting in high computational overhead, slow processing speed, and difficulty in meeting real-time requirements. In large-scale data processing scenarios, the performance of traditional models is even worse, making it impossible to efficiently perform fast image identification and batch processing. Therefore, how to design an image identification model that can ensure both high accuracy and high efficiency has become an important research direction.

[0006] (3) Poor interpretability: Deep learning-based image identification models, especially GAN-generated image identification models, mostly adopt a black-box model architecture. Due to the complex nonlinear structure and large number of parameters inside these deep neural networks, the decision-making process of the model lacks transparency, making it difficult to verify the credibility of its inference results. This not only increases the risk of the model in practical applications, but also makes the algorithm lack interpretability, failing to clearly reveal the specific reasons why fake images are identified as fake images. In scenarios with high requirements for transparency, such as law and news reporting, this lack of interpretability may become an obstacle restricting its widespread application. Therefore, how to improve the interpretability of deep learning models, so that they can effectively explain the decision-making process while maintaining high accuracy, has become a hot issue in the current field of artificial intelligence. Summary of the Invention

[0007] The purpose of this application is to provide a method and system for identifying fake images based on multimodal analysis, which can solve the problems of insufficient accuracy, low efficiency and poor interpretability of existing image identification methods.

[0008] To solve the above-mentioned technical problems, this application is implemented as follows:

[0009] In a first aspect, embodiments of this application provide a method for identifying fake images based on multimodal analysis, the method comprising: acquiring a preprocessed image;

[0010] A deep neural network for generating fake images is constructed based on multimodal image discrimination technology. The deep neural network includes: a backbone network module, a multimodal feature extraction module, an adaptive attention module, a dynamic fusion module, a classification module, and a visualization module.

[0011] A model for distinguishing real from fake images is obtained based on the deep neural network.

[0012] The preprocessed image is processed according to the real / fake image identification model to obtain image classification visualization results.

[0013] As an optional implementation of the first aspect of this application, the multimodal discrimination image generation technology is specifically multimodal fusion, which is expressed as:

[0014] ;

[0015] in, N It is the number of modes. These are the characteristics after fusion. It is the first Features of each modality The weights are determined by the adaptive attention mechanism. It is for each modal feature The nonlinear transformation function, It is a global context integration function. It is a learnable equilibrium parameter.

[0016] As an optional implementation of the first aspect of this application, the backbone network module is jointly composed of an improved ConvNext, an EfficientNet with an introduced feature recalibration mechanism, and a SwinTransformer with an introduced progressive window attention mechanism. The improved ConvNext is used to extract image depth features through depth-separable convolution; the EfficientNet with the feature recalibration mechanism is used to extract local detail features; and the Swin Transformer with the progressive window attention mechanism is used to capture global context features. The improved ConvNext will convert the original... m×m Large convolution kernels are decomposed into 1 ×m and m The ×1 operation, consisting of two consecutive convolution operations, decomposes the original 7×7 convolution kernel into three convolution kernels of sizes 3×3, 1×7, and 7×1, respectively. The mathematical expression is as follows:

[0017] ;

[0018] in, Represents the input feature map, This represents the output feature map after the improved convolution operation. This indicates convolution processing. m Represents a constant.

[0019] As an optional implementation of the first aspect of this application, the EfficientNet feature recalibration mechanism adds an adaptive feature adjustment branch to the standard MBConv module. This branch is used to dynamically adjust the importance of different channels through a lightweight channel attention network. The mathematical expression is as follows:

[0020] ;

[0021] in, Indicates the first Characteristics of each channel This indicates the first [characteristic] after adjustment by the feature recalibration mechanism. Characteristics of each channel GAP(F) Indicates the first The result after global average pooling of the channel features. and For a learnable parameter matrix, To control the scaling factor of the adjustment intensity, express Activation function.

[0022] As an optional implementation of the first aspect of this application, the progressive window attention mechanism's SwinTransformer includes multiple Swin Transformer blocks, each Swin Transformer block specifically including:

[0023] The LayerNorm module is used to normalize the input to obtain a normalized input sequence;

[0024] Window multi-head self-attention is used to perform self-attention weight calculation within a fixed local window based on the normalized input sequence to obtain an attention score;

[0025] Window-shifting multi-head self-attention is used to move windows at different stages, enabling information exchange between adjacent areas and overcoming the limitations of simple local attention.

[0026] The self-attention weights incorporate a local structure-aware term, and their mathematical expression is:

[0027] ;

[0028] in, Indicates a query. Indicates key, The values ​​represent different representations of the feature map. The dimension representing the key. The dynamic importance adjustment factor of the value is indicated. Represents a lightweight function. Indicates transpose calculation. This indicates the calculation of attention score. express Activation function.

[0029] As an optional implementation of the first aspect of this application, the multimodal feature extraction module specifically includes: a texture feature processing module, a frequency domain feature processing module, an edge feature processing module, and a geometric feature processing module, wherein the texture feature processing module, the frequency domain feature processing module, the edge feature processing module, and the geometric feature processing module are connected in parallel, wherein...

[0030] The texture feature processing module is used to process the input image to obtain the texture information of the input image;

[0031] The frequency domain feature processing module is used to generate a frequency domain representation of the input image after performing grayscale conversion, windowing, and fast Fourier transform processing on the input image.

[0032] The edge feature processing module is used to perform edge feature extraction processing on the input image to obtain the edge features of the input image;

[0033] The geometric feature processing module is used to calculate the normalized central moments of the binarized target region to obtain multiple geometric shape feature matrices.

[0034] As an optional implementation of the first aspect of this application, the visualization module is used to generate a heatmap of the image, the mathematical expression of which is:

[0035] ;

[0036] ;

[0037] in, These are elements in the convolutional feature map. Indicates the number of channels. and These represent the coordinates of the feature map in the spatial dimension, used to locate the numerical values ​​at specific locations in the feature map, reflecting the feature response at that location in the corresponding channel. Indicates the output category For the last convolutional feature map The gradient weight coefficients. It is determined by a series of parameters related to the feature map elements. The gradient is obtained through gradient calculation and is used to measure the performance in the category. In the judgment, the first The degree of contribution of different positions of the feature maps in each channel to the final result. Represents a heat map. express Activation function Indicates the output category The corresponding output results.

[0038] Secondly, embodiments of this application provide a fake image identification system based on multimodal analysis, the system comprising:

[0039] The preprocessing module is used to preprocess the image to obtain a preprocessed image;

[0040] A deep neural network module is used to construct a deep neural network for fake images. The deep neural network includes: a backbone network module, a multimodal feature extraction module, an adaptive attention module, a dynamic fusion module, a classification module, and a visualization module.

[0041] The image identification module is used to obtain a real / fake image identification model based on the deep neural network module;

[0042] Image classification visualization results are used to process the preprocessed image according to the real / fake image identification model to obtain image classification visualization results.

[0043] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0044] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0045] Compared with existing technologies, the advantages of the fake image identification method based on multimodal analysis provided by this invention are as follows:

[0046] First, high accuracy: It integrates multimodal features, including texture features, frequency domain features, edge features, and geometric features, which significantly improves the recognition accuracy in complex scenes.

[0047] Second, high efficiency: It reduces computational overhead through a lightweight classification module and dynamic fusion mechanism, and supports real-time processing.

[0048] Third, strong interpretability: The visualization module provides heatmaps, which intuitively display the basis for decision-making and enhance credibility.

[0049] Fourth, computational efficiency: The innovative progressive computing framework intelligently activates necessary modal branches based on the initial detection confidence level, realizing dynamic resource allocation. While maintaining high accuracy, it significantly reduces computing resource consumption, making the system suitable for resource-constrained real-world scenarios. Attached Figure Description

[0050] Figure 1 The flowchart illustrates a method for identifying fake images based on multimodal analysis provided in the first embodiment of the present invention.

[0051] Figure 2 This is an example diagram showing the comparison of visual heatmaps provided in the first embodiment of the present invention;

[0052] Figure 3 This diagram illustrates the structure of a fake image identification system based on multimodal analysis provided in the second embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0055] The following description, in conjunction with the accompanying drawings, details a method and system for identifying fake images based on multimodal analysis provided in this application, through specific embodiments and application scenarios.

[0056] Example 1

[0057] Please see Figure 1 This is a flowchart of a fake image identification method based on multimodal analysis provided by the present invention, including steps S1 to S4.

[0058] Step S1: Obtain the preprocessed image.

[0059] Specifically, step S1, which involves obtaining the preprocessed image, includes steps S11 to S15.

[0060] S11: Collect images.

[0061] Specifically, this invention collects 20,000 images from major social media platforms and photography platforms and performs data augmentation processing. The data augmentation processing includes random horizontal flipping, random vertical flipping, random rotation, color jittering, random cropping, and random grayscale conversion. The images are randomly cropped to a size suitable for the model input (384*384 pixels). The degradation processing includes blurring and noise addition.

[0062] S12: Crop the image.

[0063] Specifically, the random cropping method of this invention crops images to a uniform size of 384×384 pixels, thereby ensuring the consistency of input data and the full extraction of local features;

[0064] S13: Adjust the cropped image to obtain the adjusted image.

[0065] Specifically, the present invention employs color dithering technology to randomly perturb the brightness, contrast, saturation, and hue of an image to adapt to changes under different lighting and color conditions, thereby obtaining an adjusted image.

[0066] S14: Flip the adjusted image.

[0067] Specifically, the present invention uses random horizontal flipping, random vertical flipping, and random rotation to enable the adjusted image to fully simulate various angular deviations and directional changes that may occur during the actual acquisition process by transforming it at different angles and directions.

[0068] S15: Obtain the preprocessed image.

[0069] Specifically, the present invention adds Gaussian noise that conforms to a preset distribution to simulate the degradation phenomenon that may occur in the actual transmission or acquisition of the image, and normalizes and truncates the image pixel values ​​to ensure that all pixel values ​​are controlled within a predetermined range, preventing numerical overflow or oversaturation, and obtaining a preprocessed image.

[0070] In summary, step S1, which acquires preprocessed images, aims to address the inherent differences in resolution and image quality between real and fake images. These images are then further processed by the preprocessing module to obtain a dataset for image detection with uniform resolution and image quality. Data augmentation and degradation processing are then applied to both real and fake images to enrich sample diversity and improve model robustness, resulting in a more varied image dataset.

[0071] S2: Construct a deep neural network for fake images based on multimodal image generation technology. The deep neural network includes: a backbone network module, a multimodal feature extraction module, an adaptive attention module, a dynamic fusion module, a classification module, and a visualization module.

[0072] Specifically, multimodal discriminative image generation technology is essentially multimodal fusion, which can be categorized into three strategies: early fusion, mid-term fusion, and late-term fusion. This scheme employs a mid-term fusion strategy, which involves first extracting features from different modalities separately, and then integrating them through an adaptive attention mechanism and dynamic fusion. Compared to early fusion, this approach better preserves the independent characteristics of each modality, while also capturing the inter-modal relationships better than late fusion. Multimodal fusion is represented as:

[0073] ;

[0074] in, N It is the number of modes. These are the characteristics after fusion. It is the first Features of each modality The weights are determined by the adaptive attention mechanism. It is for each modal feature The nonlinear transformation function, It is a global context integration function. It is a learnable equilibrium parameter.

[0075] Furthermore, the backbone network module in step S2 is jointly composed of an improved ConvNext, an EfficientNet with a feature recalibration mechanism, and a Swin Transformer with a progressive window attention mechanism. In the backbone network module, the initial layers of ConvNext, EfficientNet, and Swin Transformer are frozen, allowing the backbone network to focus more on image features rather than retraining the network. Specifically, the improved ConvNext is used to extract image depth features through depthwise separable convolutions; the EfficientNet with its feature recalibration mechanism is used to extract local detail features; and the Swin Transformer with its progressive window attention mechanism is used to capture global contextual features.

[0076] Traditional ConvNext is a deep learning architecture based on Convolutional Neural Networks (CNNs), possessing stronger feature extraction capabilities, especially excelling in handling complex image details. Depthwise separable convolution is one of the core innovations of the ConvNext architecture, achieving a balance between computational efficiency and model performance by decomposing standard convolution operations. Specifically, depthwise separable convolution decomposes standard convolution, allowing the convolution kernel to operate independently on different channels, significantly reducing the number of model parameters and computational cost. Specifically, depthwise separable convolution decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. First, independent convolution operations are performed between channels, and then the results are combined through pointwise convolution.

[0077] Furthermore, for the depthwise convolution stage, each input channel uses a separate convolution kernel. Processing:

[0078] ;

[0079] in, Represents the first... One channel, It is the depthwise convolution kernel applied to this channel. These are the corresponding output features. It is a non-linear activation function. This indicates convolution calculation.

[0080] Next, pointwise convolution (1×1 convolution) is used to integrate the information from each channel to generate the final output feature map. :

[0081] ;

[0082] in, Indicates the number of input channels. Indicates from the input channel To output channel Pointwise convolution weights, It is a channel attention scoring function used to dynamically adjust the importance of each channel. This modification introduces an adaptive channel attention mechanism, making the effect of depthwise separable convolution more dynamic and adaptive.

[0083] Compared to standard convolution, depthwise separable convolution significantly reduces computational complexity and the number of parameters. For standard convolution, the number of input channels is... The number of output channels is The kernel size is Then the number of parameters is The number of parameters for depthwise separable convolution is only... The computational complexity was reduced by approximately times.

[0084] Considering the computational redundancy of depthwise separable convolution when processing features in different directions, this invention further improves the depthwise separable convolution in ConvNext by decomposing large convolution kernel operations to improve feature extraction efficiency. Specifically, the improved ConvNext decomposes the original... Large convolution kernels are decomposed into and Two consecutive convolution operations decompose the original 7×7 convolution kernel into three convolution kernels of sizes 3×3, 1×7, and 7×1, respectively. The mathematical expression is as follows:

[0085] ;

[0086] in, Represents the input feature map, This represents the output feature map after the improved convolution operation. This indicates convolution processing. Represents a constant.

[0087] Furthermore, the standard EfficientNet is a highly efficient convolutional neural network architecture, whose core innovation lies in the introduction of a compound scaling method. Traditionally, the depth, width, and input resolution of a network are usually adjusted separately, while compound scaling adjusts all three dimensions simultaneously through a unified coefficient, thereby improving model performance while maintaining computational efficiency. The scaling formula is as follows:

[0088] ;

[0089] in, Indicates network depth, i.e., the number of layers. This represents network width, or the number of channels. For the input image resolution, Scaling factor Indicates the global scaling factor. , and These are adaptive adjustment functions related to the content of the input image I, making the network scaling more content-aware.

[0090] In the EfficientNet architecture, the MBConv module serves as its core building block, primarily used for feature extraction. Multiple MBConv modules are stacked together to form a deep feature extraction network. Furthermore, EfficientNet introduces the Swish activation function, which demonstrates better performance on multiple tasks compared to the traditional ReLU activation function. The mathematical expression for the Swish activation function is:

[0091] ;

[0092] in, The output after activation. For the Sigmoid function, This indicates input.

[0093] The core of the MBConv module lies in efficiently extracting features and enhancing the model's expressive power. First, it increases the number of channels through dimensionality upscaling, providing richer feature information for subsequent depthwise convolutions. Next, it employs depthwise separable convolutions to perform independent convolution operations on each channel to capture local spatial information, while pointwise convolutions mix information across channels, improving feature expressiveness. To further optimize the feature extraction process, this invention introduces a Squeeze-and-Excitation (SE) module to adaptively adjust the weights of each channel. The SE module first extracts global information from the channels through global average pooling, then calculates channel weights through two fully connected layers and recalibrates the original features to enhance the expression of key features. Furthermore, to control computational complexity, the MBConv module uses 1×1 convolutions for dimensionality reduction, reducing the number of channels and thus lowering computational overhead. Simultaneously, a Dropout layer is added to randomly discard some neurons during training to prevent overfitting and improve the model's generalization ability.

[0094] To further improve the feature extraction performance of EfficientNet, this invention improves the MBConv module to obtain the EfficientNet with the feature recalibration mechanism of this invention, which introduces a feature recalibration mechanism. Specifically, based on the standard MBConv module, this invention adds an adaptive feature adjustment branch, which dynamically adjusts the importance of different channels through a lightweight channel attention network, as shown in the following mathematical expression:

[0095] ;

[0096] in, Indicates the first Characteristics of each channel This indicates the first [characteristic] after adjustment by the feature recalibration mechanism. Characteristics of each channel GAP(F) Indicates the first The result after global average pooling of the channel features. and For a learnable parameter matrix, To control the scaling factor of the adjustment intensity, express Activation function.

[0097] Furthermore, the standard Swin Transformer is a Transformer architecture specifically designed for vision tasks. Its main innovations lie in the introduction of a hierarchical structure and a sliding window mechanism. The Swin Transformer reduces computational complexity by dividing the image into multiple non-overlapping windows and performing self-attention computation within each window. In addition, the Swin Transformer enhances the model's global modeling capabilities by moving the window positions at each stage to achieve information interaction between different windows. This design enables the Swin Transformer to effectively process high-resolution images and achieve excellent performance in multiple vision tasks. The Swin Transformer architecture consists of multiple stages, each composed of multiple Swin Transformer blocks.

[0098] This invention makes targeted improvements to the standard Swin Transformer, introducing an efficient progressive window attention mechanism. Unlike the original Swin Transformer, which uses a fixed-size window at each layer, this invention employs a bottom-up gradually increasing window size strategy:

[0099] ;

[0100] in Indicates the first Layer window size, This represents the number of layers in the network, with values ​​ranging from 1 to the total number of layers. L , Base window size, This is the scaling factor.

[0101] Furthermore, the Swin Transformer block efficiently extracts and fuses local and global features through the collaborative work of multiple key components. The LayerNorm module normalizes the input, improving training stability and accelerating convergence. Subsequently, the SwinTransformer block employs multi-head self-attention within a fixed local window to accurately capture local features. To enhance global information modeling capabilities, window-shifting multi-head self-attention is introduced. By moving the window at different stages, information from adjacent regions can interact, overcoming the limitations of simple local attention. Finally, the extracted features are further processed by a multilayer perceptron (MLP) to improve feature representation and ensure efficient information flow within the deep network.

[0102] Furthermore, this invention optimizes the attention calculation mechanism within the window by introducing a local structure-aware term when calculating attention weights:

[0103] ;

[0104] in, Indicates a query. Indicates key, The values ​​represent different representations of the feature map. The dimension representing the key. The dynamic importance adjustment factor of the value is indicated. Represents a lightweight function. Indicates transpose calculation. This indicates the calculation of attention score. express Activation function.

[0105] Sliding window multi-head self-attention (SW-MSA) enables cross-window information exchange by shifting the window before calculating self-attention:

[0106] ;

[0107] Among them, CyclicShift( X ) indicates a cyclic shift operation, W-MSA indicates sliding window multi-head self-attention processing, SW-MSA ( X ) represents the output of the sliding window multi-head self-attention function.

[0108] Furthermore, the purpose of the multimodal feature extraction module in step S2 is to identify the authenticity of an image by using a feature-assisted network that extracts features from the spatial domain, frequency domain, and spectral domain.

[0109] Specifically, the multimodal feature extraction module includes: a texture feature processing module, a frequency domain feature processing module, an edge feature processing module, and a geometric feature processing module. These modules are all connected in parallel.

[0110] Specifically, the texture feature processing module performs grayscale conversion on the input image to remove color information and reduce computational complexity. Then, using an adaptive local binary mode, for each pixel's neighborhood (3×3), the grayscale value of the center pixel is compared with that of its neighboring pixels, converting it to an 8-bit binary mode and further mapping it to an LBP image within the range of 0-255. Histogram statistics are performed on the LBP image to calculate the distribution of all pixel values, thus forming a 256-dimensional texture histogram. To reduce redundant information, this invention uses a uniform mode (Uniform LBP), retaining only features with minimal mode variation, reducing the histogram dimension to 59 dimensions. To further reduce feature dimensionality and improve computational efficiency, the histogram is standardized, and then principal component analysis is applied, retaining the first few principal components, significantly reducing feature dimensionality while preserving as much of the main texture information as possible.

[0111] The mathematical expression for the adaptive local binary mode is:

[0112] ;

[0113] in, Indicates an adaptive local binary mode. This represents the weights dynamically calculated based on pixel location and image content. Indicates the center pixel coordinates. Indicates the grayscale value of the center pixel. Represents the grayscale value of neighboring pixels. Indicates the number of neighboring pixels. Represents the neighborhood radius. The symbolic function is expressed mathematically as follows:

[0114] ;

[0115] in, The input value of the sign function represents a calculated numerical value used to determine whether the value is greater than or equal to 0, thereby determining the output of the sign function.

[0116] Specifically, the frequency domain feature processing module preprocesses the input image, first converting it to grayscale, then applying a windowing operation to reduce spectral leakage and improve the accuracy of spectral calculation. It performs a Fast Fourier Transform to convert the image from the spatial domain to the frequency domain; the calculation formula is as follows:

[0117] ;

[0118] in, This is the image representation transformed to the frequency domain after Fast Fourier Transform. and These represent the number of pixels in the input image in the horizontal and vertical directions, respectively. Indicates the input image in coordinates Pixel value at that location, and These represent the horizontal and vertical coordinates of an image pixel in the spatial domain, respectively, with values ​​ranging from 0 to... and 0 to , and These are frequency coordinates in the frequency domain, representing the positions of different frequency components in the frequency domain.

[0119] Specifically, the edge feature processing module first uses the Canny edge detection method to extract the main edges, and then uses Gaussian filtering (σ=1.0) to smooth the image and reduce noise interference. Gradient magnitude and direction are calculated, and the Sobel operator is used to calculate the gradients in the horizontal and vertical directions respectively, and the gradient magnitude and angle information are synthesized. Non-maximum suppression is used to filter local maxima in the gradient direction to refine the edges. Dual thresholding (low threshold 50, high threshold 150) is used to filter strong edges and connect weak edges, ultimately generating a binary edge map. Breakpoint detection is performed to analyze edge connectivity: contour tracking extracts complete edge chains. The number of endpoints, i.e., points containing only one edge pixel in their neighborhood, is counted to assess edge integrity. Finally, an orientation histogram is constructed to analyze the gradient direction distribution of edge pixels: the gradient directions of all edge pixels are counted (range 0-180°, avoiding symmetry effects). The pixels are divided into multiple intervals by angle (e.g., 9 intervals in total, each 20°), and normalized to a probability distribution to provide a stable orientation feature representation.

[0120] Specifically, the geometric feature processing module is used to calculate the normalized central moments of the binarized target region, thereby generating 7 Hu moments. For images ,definition The order moment is:

[0121] ;

[0122] in, and is a non-negative integer used to determine the order of the moments. Corresponding to the image in Power of direction, Corresponding to the image in Power of direction, Together they determine the moment number, through different and The values ​​are used to calculate moments of different orders in order to obtain geometric feature information of the image.

[0123] When calculating the normalized central moments of the binarized target region, the mathematical expression for the central distance is:

[0124] ;

[0125] ;

[0126] in, It is the center distance, representing the image distance about the centroid. of The moment of an image pixel relative to the centroid is described by the moment of the image pixel. and Distribution in direction, It is the zero-order center distance, which is equivalent to the total number of pixels in the image. Used to standardize the center distance. It is a standardized central moment used to construct Hu invariant moments to provide a geometric feature representation of the image, and it is invariant to translation, rotation, and scaling.

[0127] Based on the mathematical expression for the center distance from the ground, the seven Hu invariant moments are specifically defined as follows:

[0128] ;

[0129] in, , , Represents the standardized central moment. From center distance and Calculation yields ( , (), used to describe the distribution characteristics of image pixels relative to the centroid in different directions, where the subscript number represents the order of the moment, such as It is the second-order central moment, which reflects the second-order feature information of the image in the x-direction during calculation. Reflects information related to second-order features in the y-direction. Information related to the first-order features that reflect the mixture in the x and y directions.

[0130] ;

[0131] in, 、 、 、 、 、 、 The standardized central moments are used to characterize the geometric features of the image from different orders and directional combinations. 、 Describes the second-order central moment. Denotes the first-order mixed central moment. 、 It represents the mixed second-order central moment, which is used in the formula calculation to construct a characteristic quantity with translation, rotation and scaling invariance.

[0132] ;

[0133] In this process, by combining these standardized central moments with different combinations and the arctangent function, the geometric feature information of the image is further extracted, so that the obtained invariant moments can stably characterize the geometric properties of the image under different transformations.

[0134] ;

[0135] The standardized central moments are combined and operated on in the formula to form a numerical description of the geometric features of the image, and have translation, rotation and scaling invariance.

[0136] ;

[0137] in, 、 、 、 、 It continues to play a role in describing the geometric distribution characteristics of images, and generates feature values ​​that reflect specific geometric properties of images and have good invariance through operations such as addition, subtraction, multiplication, division, and logarithms.

[0138] ;

[0139] This invention utilizes operations such as the sine function to further enrich the characterization of image geometric features, ensuring that the obtained invariant moments can exist stably under different image transformations and be used for feature recognition.

[0140] ;

[0141] in, 、 、 、 、 、 、 The standardized central moments, which are continuously used to describe the geometric distribution of an image, are used in calculations. Combined with the cosine function, invariant moments that can stably describe the geometric features of an image are obtained and can be used in tasks such as image recognition and classification.

[0142] The seven Hu invariant moments mentioned above are invariant to translation, rotation, and scale, and can stably describe the geometric features in an image. The mirror symmetry index is then calculated.

[0143] ;

[0144] in, SI Represents the index of mirror symmetry. and These represent the left and right mirror images respectively. Moment vector.

[0145] Furthermore, the weight allocation formula for the adaptive attention module in step S2 is as follows:

[0146] ;

[0147] in, For the first Model weights It is a learnable parameter matrix.

[0148] Furthermore, in step S2, the dynamic fusion module uses a bottleneck structure convolutional layer to compress the feature dimension to 1 / 4 of the original, and indirectly achieves feature complementarity through a cross-modal multi-head attention mechanism.

[0149] Furthermore, the classification module in step S2 outputs a binary classification result (true / false) and generates a heatmap through the visualization module to enhance the interpretability of the decision.

[0150] Step S3: Obtain a real / fake image identification model based on the deep neural network.

[0151] Specifically, this invention trains the weights of a real / fake image identification model based on a deep neural network. During the training process, when the model weights remain unchanged, the training ends and the model weights stop updating. At this point, the model weights are the optimized model weights, and the real / fake image identification model is obtained based on these optimized model weights.

[0152] Step S4: Process the preprocessed image according to the real / fake image identification model to obtain image classification visualization results.

[0153] Furthermore, the visualization module in step S2 is a key component of the multimodal fake image identification system, designed to provide interpretability support for the model's decision-making process. While deep learning models are widely used in image identification, their "black box" nature makes their decision-making process difficult to understand and trust. This invention introduces Grad-CAM++-based visualization technology to visually present the image regions of interest to the model in the form of heatmaps, generating heatmap images that can represent the authenticity of images, thereby effectively solving the problem of poor interpretability of deep learning models.

[0154] The visualization module aims to transfer the features the model focuses on onto the original image to create a heatmap, aiding in verifying the model's accuracy in distinguishing between real and fake images. The visualization module generates heatmaps based on Grad-CAM++, an improved version of Grad-weighted Class Activation Mapping. Grad-CAM++ generates class activation heatmaps by calculating weighted gradients of specific classes relative to convolutional feature maps. Compared to the original Grad-CAM, Grad-CAM++, by introducing higher-order derivatives and pixel-level weight calculations, can more accurately locate objects, especially performing better when handling multiple instances of the same class in the same image. Its formula is:

[0155] ;

[0156] ;

[0157] in, These are elements in the convolutional feature map. Indicates the number of channels. and These represent the coordinates of the feature map in the spatial dimension, used to locate the numerical values ​​at specific locations in the feature map, reflecting the feature response at that location in the corresponding channel. Indicates the output category For the last convolutional feature map The gradient weight coefficients. It is determined by a series of parameters related to the feature map elements. The gradient is obtained through gradient calculation and is used to measure the performance in the category. In the judgment, the first The degree of contribution of different positions of the feature maps in each channel to the final result. Represents a heat map. express Activation function Indicates the output category The corresponding output results. Figure 2This diagram shows a comparison example of the visualized heatmaps provided by the present invention, wherein the left side represents the original image without visualization processing, and the right side represents the heatmap obtained after visualization processing.

[0158] In summary, this invention proposes a novel hierarchical multi-scale feature integration framework that fuses multimodal information at different resolution levels. This framework processes features of each modality through a parallel feature pyramid network, and then aggregates multi-level feature representations through cross-scale connections, effectively capturing various forgery traces from micro-texture to macro-structure, significantly improving the detection capability for complex forgeries. This invention innovatively designs a context-aware meta-controller that dynamically adjusts the weight allocation of each modality feature based on the specific content features of the input image. This meta-controller learns the correlation between image content and forgery type, enabling the model to adaptively emphasize the most relevant detection modality when facing different types of forgeries. This invention specifically improves the standard Swin Transformer by introducing an efficient progressive window attention mechanism. Unlike the original Swin Transformer which uses a fixed-size window at each layer, this invention adopts a bottom-up gradually increasing window size strategy. This design allows shallow networks to focus on capturing local detail features, while deeper networks focus more on global semantic information, forming a feature extraction hierarchy from fine-grained to coarse-grained, which is particularly suitable for detecting image forgery traces at different scales.

[0159] Example 2

[0160] This invention constructs a fake image identification method based on multimodal feature fusion. This method includes extracting global and local features of the image through a backbone network module, capturing multidimensional information through a multimodal feature extraction module (texture, frequency domain, edge, and geometric features); dynamically allocating modal weights through an adaptive attention module, and integrating them into a unified representation through a dynamic fusion module; a classification module outputs a binary classification result (real / fake), and a visualization module generates a heatmap to enhance the interpretability of the decision. After preprocessing the image, this invention uses ConvNext, EfficientNet, and Swin Transformer as the backbone network and modal feature extraction modules (texture feature processing module, frequency domain feature processing module, edge feature processing module, and geometric feature processing module) to extract different features from the image and construct a multimodal feature fusion network. Through the actions of different modules in the multimodal feature fusion network, different modal features are deeply fused and the weights of different modalities are dynamically adjusted. Finally, the identification result is obtained through a classification module.

[0161] Please see Figure 3 This diagram illustrates the internal structure of a fake image identification system based on multimodal analysis provided by the present invention, comprising:

[0162] Preprocessing module 100 is used to preprocess the image to obtain a preprocessed image;

[0163] The deep neural network module 200 is used to construct a deep neural network for fake images. The deep neural network includes: a backbone network module, a multimodal feature extraction module, an adaptive attention module, a dynamic fusion module, a classification module, and a visualization module.

[0164] Image identification module 300 is used to obtain a real / fake image identification model based on the deep neural network module;

[0165] The image classification visualization result module 400 is used to process the preprocessed image according to the real and fake image identification model to obtain the image classification visualization result.

[0166] In summary, the beneficial effects of the fake image identification system based on multimodal analysis provided by this invention are as follows:

[0167] First, this invention improves the MBConv module by introducing a feature recalibration mechanism. This improvement enables the network to pay more attention to feature channels containing forgery traces, improving the detection capability of low-quality generated images, while only increasing the computational overhead by a very small amount, effectively improving the efficiency of fake image identification.

[0168] Secondly, this invention, based on sliding window multi-head self-attention (SW-MSA), improves the input feature map by shifting the window before calculating self-attention. By performing cyclic shifting, the window is moved, allowing it to cross the original boundary before calculating self-attention, thus enabling cross-window information exchange. This improvement allows the model to better preserve the local structural information of the image, which has a significant advantage in detecting local structural anomalies in real and fake images, while the increase in computational overhead is very small.

[0169] The fake image identification system based on multimodal analysis in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0170] The fake image identification system based on multimodal analysis in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0171] This application provides a fake image identification system based on multimodal analysis, which can achieve... Figures 1 to 2 The various processes implemented in the method embodiment of a fake image identification system based on multimodal analysis will not be described again here to avoid repetition.

[0172] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a fake image identification method based on multimodal analysis and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0173] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a fake image identification method based on multimodal analysis and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0174] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0175] It should be noted that in this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0177] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for fake image discrimination based on multi-modal analysis, characterized in that, The method comprises the following steps: obtaining a preprocessed image; constructing a deep neural network for false image based on a multi-modal discriminant image generation technology, the deep neural network comprising: a backbone network module, a multi-modal feature extraction module, an adaptive attention module, a dynamic fusion module, a classification module, and a visualization module; The backbone network module is jointly constituted by an improved ConvNext, an EfficientNet introducing a feature recalibration mechanism and a Swin Transformer introducing a progressive window attention mechanism, wherein the improved ConvNext is used to extract image depth features through a depth separable convolution; the EfficientNet with the feature recalibration mechanism is used to extract local detail features; and the Swin Transformer with the progressive window attention mechanism is used to capture global context features. m x m A large convolution kernel is decomposed into 1 ×m And m Two consecutive convolution operations, which decompose the original 7x7 convolution kernel into three convolution kernels with sizes of 3x3, 1x7 and 7x1, respectively, and the mathematical expression is: ； wherein, represents an input feature map, represents an output feature map after an improved convolution operation, represents a convolution process, m represents a constant; The Swin Transformer of the progressive window attention mechanism comprises a plurality of Swin Transformer blocks, and each Swin Transformer block specifically comprises: a LayerNorm module for normalizing an input to obtain a normalized input sequence; a window multi-head self-attention for performing self-attention weight calculation within a fixed local window according to the normalized input sequence to obtain an attention score; a window shift multi-head self-attention for moving the window at different stages to enable information exchange between adjacent regions and make up for the limitations of pure local attention; wherein the self-attention weight introduces a local structure perception term, and the mathematical expression is: ； wherein, denotes a query, denotes a key, denotes that the value is a different representation of the feature map, denotes the dimension of the key, denotes a dynamic importance adjustment factor for the value, denotes a lightweight function, denotes a transpose computation, denotes an attention score computation, denotes activation function; obtaining a true-false image discrimination model according to the deep neural network; processing the preprocessed image according to the true-false image discrimination model to obtain an image classification visualization result.

2. The method of claim 1, wherein the method is based on multi-modal analysis. The multi-modal discriminant image generation technology is specifically multi-modal fusion, and the multi-modal fusion is represented as: ； in, N It is the number of modes. These are the characteristics after fusion. It is the first Features of each modality The weights are determined by the adaptive attention mechanism. It is for each modal feature The nonlinear transformation function, It is a global context integration function. It is a learnable equilibrium parameter. 3.The method of claim 1, wherein, The feature recalibration mechanism of the EfficientNet adds an adaptive feature adjustment branch based on the standard MBConv module, which is used to dynamically adjust the importance of different channels through a lightweight channel attention network, and the mathematical expression is: ； wherein, represents the feature of the th channel, represents the feature of the th channel after adjustment by the feature re-calibration mechanism, GAP(F) represents the result after global average pooling of the th channel feature, and is a learnable parameter matrix, is a scaling factor to control the adjustment strength, represents an activation function.

4. The method of claim 1, wherein the method is based on multi-modal analysis. The multi-modal feature extraction module specifically comprises a texture feature processing module, a frequency domain feature processing module, an edge feature processing module, and a geometric feature processing module, which are connected in parallel, wherein the texture feature processing module is used to process an input image to obtain texture information of the input image; the frequency domain feature processing module is used to generate a frequency domain input image representation after performing gray scale conversion, windowing, and fast Fourier transform processing on the input image; the edge feature processing module is used to perform edge feature extraction processing on the input image to obtain edge features of the input image; the geometric feature processing module is used to perform normalized central moment calculation on a binarized target region to obtain a plurality of geometric shape feature matrices.

5. The method as claimed in claim 1, wherein the method is based on multi-modal analysis. The visualization module is used to generate a heat map of the image, and the mathematical expression is: ； ； in, These are elements in the convolutional feature map. Indicates the number of channels. and These represent the coordinates of the feature map in the spatial dimension, used to locate the numerical values ​​at specific locations in the feature map, reflecting the feature response at that location in the corresponding channel. Indicates the output category For the last convolutional feature map The gradient weight coefficients are obtained through a series of features map elements. The gradient is obtained through gradient calculation and is used to measure the performance in the category. In the judgment, the first The degree of contribution of different positions of the feature maps in each channel to the final result. Represents a heat map. express Activation function Indicates the output category The corresponding output results.

6. A fake image discrimination system based on multi-modal analysis, characterized in that, The system is used to implement the false image discrimination method based on multi-modal analysis according to any one of claims 1-5, and the system comprises: a preprocessing module for preprocessing an image to obtain a preprocessed image; a deep neural network module for constructing a deep neural network for false image, the deep neural network comprising: a backbone network module, a multi-modal feature extraction module, an adaptive attention module, a dynamic fusion module, a classification module, and a visualization module; an image discrimination module for obtaining a true-false image discrimination model according to the deep neural network module; An image classification visualization result module is configured to process the preprocessed image according to the true or false image discrimination model to obtain an image classification visualization result.

7. An electronic device, comprising: A processor, a memory, and a program or instructions stored on the memory and executable on the processor are included, and the program or instructions are executed by the processor to implement the steps of the false image discrimination method based on multi-modal analysis according to any one of claims 1-5.

8. A readable storage medium, characterized by, A program or instructions are stored on the readable storage medium, and the program or instructions are executed by the processor to implement the steps of the false image discrimination method based on multi-modal analysis according to any one of claims 1-5.

Images