Image forgery positioning system based on bimodal vision Mangbar network

Abstract

The invention discloses an image forgery positioning system based on a bimodal vision Mangbar network, which relates to the technical field of image forgery positioning and comprises a model construction unit and a forgery positioning unit. The model construction unit is used for preprocessing the to-be-detected image and acquiring an RGB modal image and a corresponding brightness gradient modal image; respectively carrying out feature extraction to obtain RGB features and corresponding brightness gradient features, carrying out fusion to obtain fusion features, and carrying out decoding through a decoder to obtain prediction maps of all stages; and the forgery positioning unit is used for receiving the prediction map of each stage, obtaining a final prediction map by adopting a voting mechanism, setting a confidence coefficient threshold value, judging an area of which the pixel value is greater than the confidence coefficient threshold value as a forgery area, superposing the final prediction map subjected to morphological operation processing with an original input image, and outputting a visual positioning result. According to the method, RGB and brightness gradient bimodal features are combined, and the positioning precision of weak-trace and high-fusion-degree forged areas is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of image forgery localization technology, and more specifically to an image forgery localization system based on a dual-modal visual Mamba network. Background Technology

[0002] With the development of deep learning technology, image forgery techniques are becoming increasingly sophisticated. Image tampering poses risks in various fields, such as removing copyright watermarks, creating fake news, and even committing perjury in court. These image tampering incidents can trigger trust crises and disrupt social order. Therefore, image tampering detection and localization are of great significance.

[0003] Image forgery localization, a core task in computer vision and media forensics, aims to detect and locate forged regions in images, such as splicing, copy-and-move, and content removal. With the widespread use of editing tools like Adobe Photoshop and generative diffusion models, the concealment of forgery traces has significantly increased, posing a serious challenge to information security and copyright protection. Existing image forgery localization methods are mainly divided into two categories: CNN-based and Transformer-based. While the former has higher computational efficiency, it is limited by the receptive field of the convolution kernel, making it difficult to capture long-range dependencies and resulting in poor performance in locating weak forgeries. Increasing the convolution kernel does not significantly improve performance. The latter relies on self-attention mechanisms to model global dependencies, but its computational complexity increases quadratically with image size, leading to excessive computational burden in dense prediction tasks. State-space models, as a recently emerging sequence modeling technique, can efficiently capture long-range dependencies. While maintaining linear computational complexity, its derivative models, such as Mamba and VMamba, provide new paths for image forgery localization, balancing global receptive field and computational efficiency. However, current technologies still have significant limitations. On the one hand, the modal features are singular, relying heavily on RGB or traditional auxiliary features, failing to effectively utilize the key forgery trace of illumination direction, and making it difficult to quantize and integrate it into deep learning models. On the other hand, feature fusion efficiency is low, failing to fully leverage the complementary advantages of multimodal features, while also exhibiting insufficient generalization ability, resulting in a sharp drop in performance in AIGC-generated images and complex forgery cases in real-world scenes. Robustness is also lacking, and localization accuracy is easily affected when faced with post-processing operations such as JPEG compression and Gaussian blur.

[0004] Therefore, how to accurately locate forgeries of various forgery types of images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an image forgery localization system based on a dual-modal visual Mamba network, which realizes accurate forgery localization of various forgery types of images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: An image forgery localization system based on a dual-modal visual Mamba network includes a model building unit and a forgery localization unit; The model building unit preprocesses the image to be detected to obtain an RGB modal image and a corresponding brightness gradient modal image; it extracts features from the RGB modal image and the corresponding brightness gradient modal image to obtain RGB features and corresponding brightness gradient features; it fuses the RGB features and corresponding brightness gradient features to obtain fused features; and it decodes the images at each stage to obtain prediction images. The fake localization unit receives prediction images from each stage, obtains the final prediction image using a voting mechanism, sets a confidence threshold, and determines regions in the final prediction image whose pixel values ​​are greater than the confidence threshold as fake regions and regions whose pixel values ​​are less than the threshold as normal regions. It then performs morphological operations on the fake regions to eliminate isolated noise points and optimize the localization boundaries. Finally, it overlays the morphologically processed final prediction image with the original input image and outputs a visualized localization result.

[0007] Preferably, the model building unit includes: The dual-modal input module resizes the RGB three-channel data of the image to be detected to obtain an RGB modal image; and preprocesses the RGB modal image to generate a brightness gradient modal image. The encoder adopts a dual-branch encoder structure, which processes the RGB modal image and the corresponding brightness gradient modal image respectively. Both branches use VMamba's tiny model as the backbone network for feature extraction to obtain RGB features and corresponding brightness gradient features. The Mamba fusion module concatenates the RGB features of the same stage with the corresponding brightness gradient features along the channel dimension to obtain the fused features; The decoder performs cross-level fusion on the fused features of each stage, outputs fully fused features, and then decodes them through transposed convolution, channel adjustment, and activation operations to obtain the prediction maps of each stage.

[0008] Preferably, the feature extraction of the encoder includes four feature extraction stages, each stage consisting of a visual state space block and a merge patch operation. The visual state space block first normalizes the modal features of the RGB modal image and the corresponding brightness gradient modal image through Layer Normalization, and then divides them into two branches. The first branch is processed by a linear layer and the SiLU activation function. The second branch is processed by a linear layer, a depthwise separable convolution, and the SiLU activation function in sequence, and then input into the 2D selective scanning module. The outputs of the two branches are fused by Element-wise Product, and then connected to the residual through a linear layer to obtain the RGB features and the corresponding brightness gradient features.

[0009] Preferably, the Mamba fusion module specifically includes: merging RGB features from the same stage With brightness gradient features By concatenating the data along the channel dimension, the fused features are obtained. ;Will Divided into two parts according to the passage. and , The SS2D module is input, and pixel-level contextual information is aggregated through global scanning to output spatial correlation features. ;right First, global average pooling is performed, then mapped to a 1D channel weight vector through a linear layer, and finally channel expansion is performed. Convert it to a 2D channel weight map ; After inputting into the SS2D module, average along the channel axis The channel weights are obtained by activating the channel with the Sigmoid function; these weights are then compared with... Perform element-wise multiplication to output channel correlation features. ;Will and After splicing by channel and performing Layer Normalization, we obtain the first... Stage fusion characteristics .

[0010] Preferably, the decoder specifically includes: for each stage Perform cross-level fusion, using 1×1 convolution to combine all layers. The number of channels has been uniformly adjusted to ,get ; for the first stage Size adaptation, Adaptive average pooling Downsampling Use identity mapping , Use bilinear interpolation Upsampling, to obtain Smooth using 3×3 convolution. ,get ;right , , , Perform Hadamard product operations and output fully fused features. Feature upsampling is achieved through transposed convolution and element-wise addition. , , ,according to calculate, ;right ,according to Calculate; then adjust the number of channels to 1 using a 1×1 convolution, and then... Generate prediction maps for each stage.

[0011] Preferably, it also includes dynamic supervision of the prediction map using the Dice loss function, whereby the total loss is defined as: ,in To create a Ground Truth mask for the forged region, This is the function for calculating Dice loss.

[0012] Preferably, the forgery localization unit uses a voting mechanism to obtain the final predicted image, sets a confidence threshold, and determines regions in the predicted image with pixel values ​​greater than the confidence threshold as forgery regions and regions with pixel values ​​less than the confidence threshold as normal regions. Morphological operations are performed on the forgery regions to eliminate isolated noise points and optimize the localization boundary. The localization result output overlays the processed final predicted image with the original input image to output a visualized localization result. At the same time, quantitative indicators are output to evaluate the localization accuracy, completing the entire image forgery localization process.

[0013] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an image forgery localization system based on a dual-modal visual Mamba network. By complementing the dual-modal features of RGB and brightness gradient, it accurately captures abnormal lighting traces in the forgery area. Combined with the deep modeling of spatial and channel correlation by the Mamba fusion module, it significantly improves the localization accuracy of weak traces and high-fusion forgery areas, and has stronger anti-interference ability for post-processing operations. Relying on the linear computational complexity architecture of the state space model, it avoids the high complexity problem of Transformer, improves computational efficiency while ensuring global feature perception, and adapts to high-resolution image scenes. The dual-modal fusion and cross-layer fusion design enhances the model's generalization ability, which can effectively deal with various types of forgery such as traditional forgery and AIGC generated forgery, and solves the pain point of sudden performance drop across scenes. At the same time, the model training does not require pre-training synthetic data, the preprocessing and inference process is simple, and the visualization results and quantitative indicators can directly meet the practical application needs of media forensics, providing more reliable and efficient technical support. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of brightness gradient generation provided by the present invention; Figure 2 A schematic diagram of the bimodal visual Mamba Network (BVM-Net) provided by this invention; Figure 3 Schematic diagram of Mamba fusion, grouped Hadamard product module, and visual state space block provided by the present invention; Figure 4 This is a schematic diagram of the SS2D module provided by the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This invention discloses an image forgery localization system based on a dual-modal visual Mamba network, comprising a model building unit and a forgery localization unit; The model building unit preprocesses the image to be detected to obtain an RGB modal image and a corresponding brightness gradient modal image; it extracts features from the RGB modal image and the corresponding brightness gradient modal image to obtain RGB features and corresponding brightness gradient features; it fuses the RGB features and corresponding brightness gradient features to obtain fused features; and it decodes the images at each stage to obtain prediction images. The fake localization unit receives prediction images from each stage, obtains the final prediction image using a voting mechanism, sets a confidence threshold, and determines regions in the final prediction image whose pixel values ​​are greater than the confidence threshold as fake regions and regions whose pixel values ​​are less than the threshold as normal regions. It then performs morphological operations on the fake regions to eliminate isolated noise points and optimize the localization boundaries. Finally, it overlays the morphologically processed final prediction image with the original input image and outputs a visualized localization result.

[0018] In one specific embodiment, the technical solution of the present invention is divided into three stages: model building, training optimization, and fake localization. The detailed process is as follows: The model building phase focuses on constructing the BVM-Net network architecture, including a bimodal input module, encoder, Mamba fusion (MF) module, decoder, and loss function design. This lays the foundation for subsequent training and localization. Specific steps include: 1. Design of Dual-Modal Input Module and Feature Preprocessing RGB modal input: The RGB three-channel data of the image to be detected is used as the basic modality. The input image size is uniformly adjusted to 416×416 (this size is used in both training and inference phases to ensure consistency in feature extraction), and is denoted as the input image. (in , ).

[0019] Brightness gradient mode generation: Reference Figure 1 The process shown is for the input image. Preprocessing is performed to generate a brightness gradient map. The process involves three steps: First, enhancing the image's contrast and color to amplify brightness differences; second, using the Sobel operator to calculate the gradient values ​​in the horizontal (Sobel_x) and vertical (Sobel_y) directions; and third, calculating the gradient magnitudes based on the horizontal and vertical gradient values ​​and integrating them to obtain a brightness gradient map. This image can quantify the rate and direction of changes in image brightness, and capture traces of abnormal lighting in forged areas.

[0020] 2. The encoder architecture adopts a dual-branch encoder structure to process RGB modal and luminance gradient modal features respectively. Both branches use VMamba's tiny model as the backbone network and contain four feature extraction stages. Each stage consists of a visual state space (VSS) block and a merging patch operation, as detailed below: VSS Block Structure: Reference Figure 3 (b) The VSS block is the core feature extraction unit of the encoder, which is derived from an improved design of VMamba. The process is as follows: the input features are first normalized by LayerNormalization and then divided into two branches. The first branch is processed by a linear layer and the SiLU activation function. The second branch is processed by a linear layer, a depthwise separable convolution, and the SiLU activation function in sequence, and then input into the 2D selective scan (SS2D) module. The outputs of the two branches are fused by Element-wise Product and then connected to the residual through a linear layer to obtain the final output of the VSS block.

[0021] SS2D module operation: Reference Figure 4 The SS2D module is responsible for capturing global spatial dependencies. The process includes: ① Scan expansion: expanding the input features into a sequence in four directions: "top left → bottom right, bottom right → top left, top right → bottom left, bottom left → top right"; ② S6 block processing: using S6 blocks (an improvement on the S4 framework with a selective mechanism) to refine the features of the sequence in each direction, and enhancing effective information and suppressing redundant information by adaptively adjusting the SSMs parameters; ③ Scan merging: restoring the processed sequences in the four directions to the original image size and outputting global dependency features.

[0022] Encoder parameter configuration: The number of VSS blocks for each of the two branches in the four stages is set to [values ​​to be filled in]. , , , The number of output feature channels in each stage is as follows: , , , (in ); Let the RGB branch be the first The stage output characteristics are The brightness gradient branch The stage output characteristics are ( Both have the same number of channels. .

[0023] 3. Mamba Fusion (MF) Module Design: The MF module is used to fuse the characteristics of bimodalities at each stage, capturing spatial and channel correlations. (Refer to...) Figure 3 (a) The specific process is as follows: Feature splicing: combining features from the same stage and By concatenating the data along the channel dimension, the fused features are obtained. .

[0024] Spatial correlation capture: Divided into two parts according to the passage. and , The SS2D module is input, and pixel-level contextual information is aggregated through global scanning to output spatial correlation features. .

[0025] Channel correlation capture: for First, global average pooling (GAP) is performed, then mapped to a 1D channel weight vector through a linear layer, and finally, channel expansion is performed. Convert it to a 2D channel weight map ; After inputting into the SS2D module, the average value is calculated along the channel axis ( The channel weights are then activated by the Sigmoid function and obtained; these weights are then compared with... Perform Element-wise Multiplication to output channel correlation features. .

[0026] Feature fusion and standardization: and After splicing by channel and performing Layer Normalization, we obtain the first... Phase final fusion characteristics .

[0027] 4. Decoder Architecture Design: The decoder achieves cross-layer feature fusion through grouped Hadamard product (GHP) modules and transposed convolutions to generate pixel-level forgery localization maps. (Refer to...) Figure 2 and Figure 3 (c) The specific process is as follows: GHP module operation: for 4 stages Perform cross-level fusion: First, use a 1×1 convolution to merge all layers. The number of channels has been uniformly adjusted to ,get The second step is to... stage Size adaptation ( Adaptive average pooling Downsampling Use identity mapping , Use bilinear interpolation Upsampling), to obtain The third step is to smooth the surface using a 3×3 convolution. ,get Fourth step, for , , , Perform Hadamard product operations and output fully fused features. .

[0028] Prediction graph generation: Feature upsampling is achieved through transposed convolution (DeConv_4×4) and element-wise addition. , , ,according to calculate( );right ,according to Calculate; then adjust the number of channels to 1 using a 1×1 convolution, and then... ( )or Generate a prediction map for 4 stages ( , , , All of them are 416×416 in size.

[0029] 5. Loss Function Design: To address the issues of low pixel proportion in forged regions and imbalanced positive and negative samples, the Dice loss function is used to dynamically supervise the four prediction images. The total loss is defined as: ,in To create a Ground Truth mask for the forged region, The Dice loss calculation function is used to measure the difference between the predicted graph and the actual graph. The degree of overlap.

[0030] The training and optimization phase involves configuring the dataset, setting hyperparameters, and training the model to enable BVM-Net to learn image forgery features. Specific steps include: The dataset configuration and partitioning utilize multiple dataset types to verify the model's generalization ability, including: ① traditional benchmark datasets (CASIA, NIST); ② AIGC datasets (AutoSplice, DID); ③ real-world scene datasets (IMD, Wild). Dataset partitioning follows these principles: CASIA and NIST are partitioned using existing standards (e.g., CASIA training set: 5123 images; test set: 921 images); other datasets are partitioned into training and test sets in an 8:2 ratio. The training set does not contain pre-trained synthetic fake data; it is trained only with real / semi-realistic fake data to ensure the model's adaptability to real-world scenarios.

[0031] The training hyperparameters were set based on the PyTorch framework, using an NVIDIA GeForce RTX 4090 graphics card for training. The hyperparameters were configured as follows: Adam optimizer was selected, and the initial learning rate was set to [value missing]. The batch size is 8, and the maximum number of training epochs is 100. During training, the input images are all resized to 416×416, and data augmentation strategies such as random flipping and brightness perturbation are used to improve the robustness of the model.

[0032] Model training and parameter saving using training set images and their corresponding parameters As input, iteratively train the model according to the process of "forward propagation to calculate the prediction graph and total loss → backpropagation to update network parameters". After each round of training, the model performance is evaluated using a validation set (using F1-Score and IoU as indicators), and the model parameters with the best performance on the validation set are saved for subsequent fake localization inference.

[0033] In the forgery localization stage, the trained BVM-Net model is used to detect and localize forgery regions in the input image. Specific steps include: Input image preprocessing: The image to be detected is preprocessed as follows: First, the image size is resized to 416×416 to maintain consistency with the training stage; Second, if the image is a single-channel grayscale image, it is automatically expanded into a three-channel RGB image (the three channels have the same values); Third, the image pixel values ​​are normalized (e.g., mapped to the [0,1] interval) to eliminate the influence of numerical range differences on model inference.

[0034] Dual-modal feature extraction and fusion

[0035] The preprocessed RGB image is input into the RGB branch of the encoder, and after four stages of VSS block and merge patch operations, the features of each stage are output. , , , .

[0036] The preprocessed image is processed according to the workflow of "contrast enhancement → Sobel operator calculation → gradient magnitude integration" to generate a brightness gradient map, which is then input into the brightness gradient branch of the encoder and output as features at each stage. , , , .

[0037] Will be in the same stage and Input MF module, output fused features , , , .

[0038] Fake Region Prediction and Post-processing

[0039] The four stages The input decoder GHP module, after cross-level fusion and transposed convolution operations, generates four prediction maps. , , , .

[0040] Post-processing of the four prediction images: First, a voting mechanism (e.g., taking the average pixel value of the four prediction images) is used to obtain the final prediction image; Second, a confidence threshold (e.g., 0.5) is set, and areas in the prediction image with pixel values ​​greater than the threshold are identified as fake areas, while areas with pixel values ​​less than the threshold are identified as normal areas; Third, morphological operations (e.g., dilation and erosion) are performed on the fake areas to eliminate isolated noise points and optimize the localization boundary.

[0041] The localization output overlays the processed final prediction image onto the original input image (e.g., marking the fake area in red) to output a visualized localization result; at the same time, it outputs quantitative indicators (F1-Score, IoU) to evaluate the localization accuracy, thus completing the entire image forgery localization process.

[0042] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The methods disclosed in the embodiments are described simply because they correspond to the methods disclosed in the embodiments; relevant parts can be found in the method section.

[0043] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image forgery localization system based on a dual-modal visual Mamba network, characterized in that, Includes model building units and fake location units; The model building unit preprocesses the image to be detected to obtain an RGB modal image and a corresponding brightness gradient modal image; it extracts features from the RGB modal image and the corresponding brightness gradient modal image to obtain RGB features and corresponding brightness gradient features; it fuses the RGB features and corresponding brightness gradient features to obtain fused features; and it decodes the images at each stage to obtain prediction images. The fake localization unit receives prediction images from each stage, obtains the final prediction image using a voting mechanism, sets a confidence threshold, and determines regions in the final prediction image whose pixel values ​​are greater than the confidence threshold as fake regions and regions whose pixel values ​​are less than the threshold as normal regions. It then performs morphological operations on the fake regions to eliminate isolated noise points and optimize the localization boundaries. Finally, it overlays the morphologically processed final prediction image with the original input image and outputs a visualized localization result.

2. The image forgery localization system based on a dual-modal visual Mamba network according to claim 1, characterized in that, The model building unit includes: The dual-modal input module resizes the RGB three-channel data of the image to be detected to obtain an RGB modal image; and preprocesses the RGB modal image to generate a brightness gradient modal image. The encoder adopts a dual-branch encoder structure, which processes the RGB modal image and the corresponding brightness gradient modal image respectively. Both branches use VMamba's tiny model as the backbone network for feature extraction to obtain RGB features and corresponding brightness gradient features. The Mamba fusion module concatenates the RGB features of the same stage with the corresponding brightness gradient features along the channel dimension to obtain the fused features; The decoder performs cross-level fusion on the fused features of each stage, outputs fully fused features, and then decodes them through transposed convolution, channel adjustment, and activation operations to obtain the prediction maps of each stage.

3. The image forgery localization system based on a dual-modal visual Mamba network according to claim 2, characterized in that, The encoder's feature extraction includes four feature extraction stages, each consisting of a visual state space block and a merge patch operation. The visual state space block first normalizes the modal features of the RGB modal image and the corresponding brightness gradient modal image through Layer Normalization, and then divides them into two branches. The first branch is processed through a linear layer and the SiLU activation function. The second branch is processed through a linear layer, a depthwise separable convolution, and the SiLU activation function in sequence, and then input into the 2D selective scanning module. The outputs of the two branches are fused by Element-wise Product, and then connected to the residual through a linear layer to obtain the RGB features and the corresponding brightness gradient features.

4. The image forgery localization system based on a dual-modal visual Mamba network according to claim 2, characterized in that, The Mamba fusion module specifically includes: combining RGB features from the same stage. With brightness gradient features By concatenating the data along the channel dimension, the fused features are obtained. ;Will Divided into two parts according to the passage. and , The SS2D module is input, and pixel-level contextual information is aggregated through global scanning to output spatial correlation features. ;right First, global average pooling is performed, then mapped to a 1D channel weight vector through a linear layer, and finally channel expansion is performed. Convert it to a 2D channel weight map ; After inputting into the SS2D module, average along the channel axis The channel weights are obtained by activating the channel with the Sigmoid function; these weights are then compared with... Perform element-wise multiplication to output channel correlation features. ;Will and After splicing by channel and performing Layer Normalization, we obtain the first... Stage fusion characteristics .

5. The image forgery localization system based on a dual-modal visual Mamba network according to claim 4, characterized in that, The decoder specifically includes: for each stage Perform cross-level fusion, using 1×1 convolution to combine all layers. The number of channels has been uniformly adjusted to ,get ; for the first stage Size adaptation, Adaptive average pooling Downsampling Use identity mapping , Use bilinear interpolation Upsampling, to obtain Smooth using 3×3 convolution. ,get ;right , , , Perform Hadamard product operations and output fully fused features. Feature upsampling is achieved through transposed convolution and element-wise addition. , , ,according to calculate, ;right ,according to Calculate; then adjust the number of channels to 1 using a 1×1 convolution, and then... Generate prediction maps for each stage.

6. The image forgery localization system based on a dual-modal visual Mamba network according to claim 5, characterized in that, It also includes dynamic supervision of the prediction graph using the Dice loss function, with the total loss defined as: ,in To create a Ground Truth mask for the forged region, This is the function for calculating Dice loss.

7. The image forgery localization system based on a dual-modal visual Mamba network according to claim 1, characterized in that, The forgery localization unit uses a voting mechanism to obtain the final predicted image. A confidence threshold is set, and regions with pixel values ​​greater than the confidence threshold in the predicted image are identified as forgery regions, while regions with pixel values ​​less than the confidence threshold are identified as normal regions. Morphological operations are performed on the forgery regions to eliminate isolated noise points and optimize the localization boundary. The localization result output overlays the processed final predicted image with the original input image to output a visualized localization result. At the same time, quantitative indicators are output to evaluate the localization accuracy, completing the entire image forgery localization process.

Images