Image forgery localization detection method based on collaborative difference optimization and multi-modal perception

Abstract

The present application relates to a kind of based on cooperative difference optimization and multimodal perception's image forgery positioning detection method, belong to computer vision and multimedia forensics technical field.It includes the following steps: obtaining RGB image carries out random data enhancement;Noise extractor is used to extract noise map;RGB image and noise map are generated multi-scale feature by pre-trained cross-modal information encoder;The multi-scale feature input abnormal encoder, the abnormal encoder includes multilayer perception, linear fusion block and linear prediction block, after multilayer perception processing obtains multi-layer mapping feature and outputs initial prediction map;Original RGB image is input into ViT model and extracts global feature, which is input into cross-scale feature enhancement module with multi-layer mapping feature to interact optimization and obtain refined positioning map;Fusion initial prediction map and refined positioning map, obtain final prediction map;Iterative optimization is carried out by loss function.The present application can improve the detection sensitivity of different size tampering area.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and multimedia forensics, and particularly relates to an image forgery positioning detection method based on collaborative difference optimization and multi-modal perception. BACKGROUND

[0002] With the rapid development of digital technology, image tampering has become a ubiquitous and complex challenge. The tampering process involves non-original modifications to digital images, including cutting and splicing, adding or deleting elements, color and brightness adjustment, and image synthesis, among other types. These operations can be used not only for legal purposes such as artistic creation, but also can be misused to spread false information. In recent years, deep learning technology has made significant progress in image forgery positioning, using convolutional neural networks as feature encoders to improve detection capabilities. However, existing methods rely on the design of advanced visual tasks, making it difficult to effectively separate the feature differences between forged and real regions, leading to false positives and missed detections. The root cause lies in the fact that tampering detection needs to prioritize local semantic-agnostic clues (such as noise and texture) over semantic content, while traditional models cannot adequately represent these subtle traces, limiting detection accuracy and robustness.

[0003] The core challenge of tampering detection lies in learning discriminative features between regions, requiring the extraction of long-range semantic information from a global perspective to identify inconsistencies. For example, analyzing the relationship between objects and backgrounds can enhance sensitivity to tampering, but existing methods rely on single-scale features or single-modal representations, which cannot capture the synergistic effects of multiple modalities (such as noise statistics and edge texture). Tampered regions in images often exhibit noise characteristics at the boundaries, and single-modal strategies struggle to match these subtle differences, resulting in insufficient detection sensitivity. Therefore, there is an urgent need for an innovative strategy that combines multi-modal perception with collaborative difference optimization, breaking through traditional limitations and achieving more accurate tampering positioning. SUMMARY

[0004] To solve the above problems, the application provides an image forgery positioning detection method based on collaborative difference optimization and multi-modal perception.

[0005] To achieve the above purpose, the following technical solutions are implemented:

[0006] The application provides an image forgery positioning detection method based on collaborative difference optimization and multi-modal perception, comprising the following steps:

[0007] An RGB image to be detected is obtained, and random data augmentation is performed on the RGB image;

[0008] The RGB image is extracted by a noise extractor to obtain a noise image; the RGB image and the noise image are input into a pre-trained cross-modal information encoder CMX to obtain multi-scale features;

[0009] The multi-scale features are processed by an anomaly encoder to obtain an initial prediction map; the anomaly encoder includes a multi-layer perception MLP, a linear fusion block, and a linear prediction block; the multi-scale features are processed by the multi-layer perception MLP to obtain multi-layer mapping features;

[0010] The RGB image X is input into a pre-trained ViT model to extract global features of the image; the global features and the multi-layer mapping features are input into a cross-scale feature enhancement module for interactive optimization to obtain a refined positioning map;

[0011] The initial prediction map and the refined positioning map are fused to obtain a final positioning map;

[0012] The above process is iteratively optimized by a loss function.

[0013] Further, the random data augmentation includes a random tampering operation based on semantic perception, a basic image transformation, and an imaging process simulation.

[0014] Further, the RGB image X is processed by a pre-trained Noiseprint++ noise extractor, an Srm noise extractor, and a Bayar noise extractor to obtain a first noise auxiliary representation , a second noise auxiliary representation , and a third noise auxiliary representation ; the first noise auxiliary representation , the second noise auxiliary representation , and the third noise auxiliary representation are fused by a fusion module FM to obtain a mixed feature ; the RGB image X and the mixed feature are input into a pre-trained cross-modal information encoder CMX to obtain a first feature , a second feature , a third feature , and a fourth feature ; the cross-modal information encoder CMX adopts a cross-attention mechanism to establish a channel attention correlation between the RGB image X and the mixed feature .

[0015] Further, the linear fusion block includes a convolution block, a batch normalization, and a ReLU activation function; the linear prediction block includes a convolution block, a Dropout layer, and an up-sampling layer.

[0016] The first feature , the second feature , the third feature , and the fourth feature After weighted processing by a multi-layer perception (MLP), a multi-layer mapping feature is obtained , After inputting the multi-layer mapping feature to a linear fusion block for fusion, a fusion feature is obtained; the fusion feature is subjected to a linear prediction block to obtain an initial prediction map .

[0017] Further, the global feature and the multi-layer mapping feature are input to a cross-scale feature enhancement module to obtain a refined positioning map Y2, and the specific process is as follows:

[0018] The first mapping feature , the second mapping feature , the third mapping feature , and the fourth mapping feature are divided into a local group and a regional group according to their receptive fields; then difference operation is performed with the global feature , and the difference result is multiplied element by element with the corresponding mapping feature to obtain an enhanced feature , ; the formula is as follows:

[0019] ,

[0020] Wherein, represents element-by-element multiplication operation; represents difference operation; the enhanced feature is fused with the global feature to obtain local group fusion feature and regional group fusion feature ; the formula is as follows:

[0021] ,

[0022] ,

[0023] Wherein, represents channel dimension splicing; , , , respectively represent the first enhanced feature, the second enhanced feature, the third enhanced feature, and the fourth enhanced feature; the local group fusion feature and the regional group fusion feature After processing, the refined positioning map Y2 is obtained. The mask G is used for training and optimization.

[0024] Furthermore, the initial prediction map The final prediction map is obtained by element-wise multiplication with the refined localization map Y2. .

[0025] Furthermore, regarding the final predicted map For each pixel in the texture vector, calculate its absolute difference from its neighboring pixels to form a texture vector. , ;in, Indicates the final prediction graph The i-th pixel in the neighborhood of the i-th pixel Texture vector of pixels; Indicates the final prediction graph The i-th pixel in the neighborhood of the i-th pixel A texture vector of 1 pixel, , , This represents the prediction result for a pixel. Indicates the middle pixel. Indicates the middle pixel The j-th pixel in the neighborhood, This represents the k×k pixels in the neighborhood of the i-th pixel;

[0026] For the first noise auxiliary representation Second noise auxiliary representation and third noise auxiliary representation Texture vectors of the three noise modalities are calculated to obtain the texture vectors of the noise images. The formula is expressed as follows:

[0027] ,

[0028] ,

[0029] in, Represents the neighborhood of the i-th pixel under the m-th noise mode. Texture vector of pixels; Represents an exponential function; Indicates hyperparameters; This represents the texture vector of the i-th pixel under the m-th noise mode; This represents the texture vector of the j-th pixel under the m-th noise mode; Indicates the number of noise modal image types; Represents the square of the Euclidean distance;

[0030] The joint consistency loss function is represented as follows:

[0031]

[0032] wherein, represents a binary boundary mask, represents a joint consistency loss function.

[0033] Further, the binary boundary mask is obtained by converting the final prediction map into a binary label to distinguish foreground and background, obtaining a binary matrix L; and extracting a boundary mask in the binary matrix L through an erosion-dilation technique, obtaining the binary boundary mask .

[0034] Further, the loss function in step S6 further includes a weighted cross-entropy loss function and a Dice loss function.

[0035] Further, the total loss function is , wherein, , , respectively represent a weight coefficient of the weighted cross-entropy loss function , a weight coefficient of the Dice loss function , and a weight coefficient of the joint consistency loss function .

[0036] The present application has the following advantages:

[0037] The present application breaks through the limitations of traditional single-scale feature analysis by using the visual transformer to capture long-distance dependencies and the innovative cross-scale feature enhancement module, which can significantly improve the detection sensitivity of different size tampering regions, especially for complex splicing, copy-move and other operations. The cross-modal information fusion module creatively integrates edge feature and noise feature analysis, realizes "double verification" of tampering traces by establishing feature consistency guidance between noise maps, effectively solves the false detection problem of traditional single-modal methods (such as relying only on noise or only texture analysis), and first realizes dynamic collaborative optimization of edge and noise features, creating a new paradigm of "cross-feature collaborative analysis", providing an expandable technical framework for the image forensics field, and being compatible with other modal features such as spectral features and deep learning fingerprints in the future. Experimental data show that the scheme surpasses the prior art in core indicators such as F1 score. BRIEF DESCRIPTION OF DRAWINGS

[0038] ​The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain the application, and do not limit the application.

[0039] Figure 1 Flow chart of steps of the method of the present application;

[0040] Figure 2 Comparison chart of edge refinement visualization of the method of the present application and the baseline model;

[0041] Figure 3 Comparison chart of false positive and false negative visualization of the method of the present application and the existing method;

[0042] Figure 4 Comparison chart of visualization of the method of the present application and the existing method. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0044] Embodiment 1

[0045] In this embodiment, as shown in the following table, the present application provides an image forgery positioning detection method based on collaborative difference optimization and multi-modal perception, and the specific steps include: Figure 1

[0046] Step one, obtaining the RGB image to be detected, and performing random data enhancement on the RGB image.

[0047] ​Specifically, the random data augmentation includes a semantic-aware random tampering operation, a basic image transformation, and an imaging process simulation; the semantic-aware random tampering operation includes: RandomCopyMove: performing a copy-move operation in a random region of the image to simulate common splicing tampering; RandomInpainting: using a deep generative model to perform content inpainting on a randomly selected region. The basic image transformation includes: HorizontalFlip / VerticalFlip: horizontal / vertical flip enhancement; RandomBrightnessContrast: random adjustment of brightness and contrast. The imaging process simulation includes: GaussianBlur: Gaussian blur (σ=0.2); RandomScale: random scaling (scale factor=0.5); RandomCrop: random cropping (cropping ratio=1); ImageCompression: JPEG compression (quality factor=0.5).

[0048] Step two, extracting noise information from the RGB image through a noise extractor to obtain a noise map; inputting the RGB image and the noise map into a pre-trained cross-modal information encoder CMX to obtain multi-scale features.

[0049] Specifically, the RGB image X is processed by a pre-trained Noiseprint++ noise extractor, an Srm noise extractor, and a Bayar noise extractor to obtain a first noise auxiliary representation , a second noise auxiliary representation , and a third noise auxiliary representation ; the first noise auxiliary representation , the second noise auxiliary representation , and the third noise auxiliary representation are fused by a fusion module FM to obtain a mixed feature ; the fusion module FM includes a first ConvBlock module, a second ConvBlock module, and a third ConvBlock module; each of the three ConvBlock modules includes a first convolutional layer, a second convolutional layer, and a third convolutional layer, and each of the first convolutional layer, the second convolutional layer, and the third convolutional layer includes a two-dimensional convolutional layer Conv2d, a two-dimensional batch normalization layer BatchNorm2d, and a ReLU activation function; the output channel numbers of the first convolutional layer, the second convolutional layer, and the third convolutional layer are 24, 48, and 96, respectively; the first noise auxiliary representation , the second noise auxiliary representation , and the third noise auxiliary representation The three auxiliary representations are input into 1x1 convolution units respectively to restore the channels to 3, to obtain three channel-restored auxiliary representations; the three channel-restored auxiliary representations are spliced in the channel dimension to obtain a mixed feature ;

[0050] The RGB image X and the mixed feature are input into a pre-trained cross-modal information encoder CMX to obtain a first feature , a second feature , a third feature , and a fourth feature ; the cross-modal information encoder CMX adopts a cross attention mechanism to establish a channel attention correlation between the RGB image X and the mixed feature .

[0051] Step three, processing the multi-scale feature through an anomaly encoder to obtain an initial prediction map; the anomaly encoder includes a multi-layer perceptron MLP, a linear fusion block, and a linear prediction block; the multi-scale feature passes through the multi-layer perceptron MLP to obtain a multi-layer mapping feature.

[0052] Specifically, the linear fusion block includes a 1x1 convolution block, batch normalization, and a ReLU activation function; the linear prediction block includes a 1x1 convolution block, a Dropout layer, and an upsampling layer.

[0053] The first feature , the second feature , the third feature , and the fourth feature are weighted processed through the multi-layer perceptron MLP to obtain a multi-layer mapping feature , , represents the first mapping feature. represents the second mapping feature. represents the third mapping feature. represents the fourth mapping feature; the formula is as follows:

[0054] ,

[0055] ,

[0056] ,

[0057] wherein, represents a convolution operation with a convolution kernel size of . represents a convolution operation with a kernel size of ; represents a batch normalization operation; represents the first processing of the i-th feature; represents the second processing of the i-th feature; represents a Sigmoid activation function; represents an element-wise multiplication operation; represents the i-th feature, ; represents the first mapped feature, represents the second mapped feature, the third mapped feature, represents the fourth mapped feature;

[0058] After inputting the multi-layer mapped features to a linear fusion block for fusion, a fused feature is obtained; the fused feature passes through a linear prediction block to obtain an initial prediction map . The specific process is as follows: the input is four feature maps of different scales , , , , which respectively represent multi-level features from shallow to deep, wherein the shallow features contain rich spatial details, and the deep features have stronger semantic information. Each feature first undergoes linear transformation for uniform mapping of the channel dimension. Then, the processed features are spliced in the order from deep to shallow to form a tensor that fuses multi-scale information, and then pass through a linear fusion block for channel dimension reduction and feature integration, outputting a fused feature. Finally, a linear prediction block is used to predict the category of each pixel to obtain an initial prediction map . The whole process fuses hierarchical features, which not only retains the high-resolution details of the shallow layer, but also combines the semantic information of the deep layer, thereby improving the segmentation accuracy.

[0059] Step four, input the RGB image X into the pre-trained ViT model to extract the global feature of the image; input the global feature and the multi-layer mapped feature into the cross-scale feature enhancement module for interactive optimization to obtain a refined positioning map.

[0060] Specifically, input the global feature and the multi-layer mapped feature into the cross-scale feature enhancement module to obtain a refined positioning map Y2, and the specific process is as follows:

[0061] The first mapped feature , the second mapped feature , the third mapped feature , and the fourth mapped feature are divided into local groups according to their receptive fields ) and region group ( ); then respectively with global feature difference operation, the difference result is multiplied with the corresponding mapping feature element by element, to obtain enhanced feature , ; the formula is as follows:

[0062] ,

[0063] wherein, indicates the difference operation; the enhanced feature is fused with the global feature to obtain local group fusion feature and region group fusion feature ; the formula is as follows:

[0064] ,

[0065] ,

[0066] wherein, indicates splicing along the channel dimension; , , , respectively indicate the first enhanced feature, the second enhanced feature, the third enhanced feature and the fourth enhanced feature; the local group fusion feature and the region group fusion feature are processed to obtain the refined positioning map Y2, and the formula is as follows:

[0067] ,

[0068] wherein, indicates aggregation along the channel dimension; the refined positioning map is trained and optimized by the mask G, and the joint loss function is as follows:

[0069] ,

[0070] wherein , respectively are Dice loss and IOU loss, indicates the joint loss function, and G indicates the mask.

[0071] Step five, the initial prediction map and the refined positioning map Y2 are multiplied element by element to obtain the final prediction map .

[0072] Step six, the above process is iteratively optimized by the loss function.

[0073] Specifically, for each pixel in the final prediction map , the absolute difference with the pixels in its neighborhood is calculated, forming a texture vector , ; where, denotes the texture vector of the th pixel in the neighborhood of the th pixel in the final prediction map denotes the texture vector of the th pixel in the neighborhood of the th pixel in the final prediction map , , denotes the prediction result of the pixel, denotes the intermediate pixel, denotes the th pixel in the neighborhood of the intermediate pixel, denotes the k x k pixels in the neighborhood of the

[0074] Texture vector calculation is performed on the three noise modal images of the first noise auxiliary representation , the second noise auxiliary representation , and the third noise auxiliary representation , obtaining noise image texture vectors , which are expressed as follows:

[0075] ,

[0076] ,

[0077] where, denotes the texture vector of the th pixel in the neighborhood of the th pixel in the mth noise modal image; denotes the exponential function; denotes the hyperparameter; denotes the texture vector of the th pixel in the mth noise modal image;

[0078] denotes the texture vector of the th pixel in the

[0079] mth noise modal image; ,

[0080] where, denotes the binary boundary mask, denotes a joint consistency loss function; the binary boundary mask The acquisition process is as follows:

[0081] The final prediction map is converted into a binary label , distinguishing foreground and background, to obtain a binary matrix L, which is expressed as follows:

[0082] ,

[0083] wherein, denotes the pixel in the i-th row and the j-th column; the binary boundary mask is extracted in the binary matrix L through an erosion-dilation technique .

[0084] Specifically, the loss function further includes a weighted cross-entropy loss function and a Dice loss function:

[0085] The formula of the weighted cross-entropy loss function is as follows:

[0086] ,

[0087] wherein, denotes the weighted cross-entropy loss; denotes the pixel i in the mask G; denotes the pixel i in the RGB image X; , respectively denote a sample weight and a positive class weight, which are respectively set to 0.5 and 2.5, for processing sample data imbalance; denotes a logarithmic function, and N denotes the total number of pixels in the final prediction map . The loss function is optimized through the classification error of each pixel, but is sensitive to edge blur.

[0088] The formula of the Dice loss function is as follows:

[0089] ,

[0090] wherein, denotes the Dice loss function; denotes a smoothing coefficient.

[0091] Specifically, the total loss function is , wherein, , , respectively denote a weight coefficient of the weighted cross-entropy loss function , a weight coefficient of the Dice loss function , and a weight coefficient of the joint consistency loss function The weighting coefficients, , , , Set them to 0.3, 0.5, and 0.2 respectively.

[0092] Example 2

[0093] This embodiment evaluates the performance of the proposed method in the task of locating image tampering. The present invention uses an optimal threshold and a fixed threshold of 0.5 to report F1 scores for comparison. As shown in Table 1, this table presents comparison data between the proposed method and other methods, with experiments conducted on the Columbia, Coverage, CocoGlide, CasiaV1, and Nist16 datasets. Bold text indicates the highest performance, and underlined text indicates the second highest.

[0094] Table 1. Comparison of experimental data between the method of the present invention and existing methods.

[0095]

[0096] Figure 2 This diagram illustrates the difference in edge refinement between the model of this invention and the TruFor model (baseline model). In the figure, the first row shows the manipulated image, followed by the edge image refined by the baseline model, the edge image refined by this method, and the mask. It can be seen that this method highlights the details and completeness of the located region better than the baseline model in edge refinement. For example, in the first column, this method shows more complete edge refinement than the baseline model. In the second column, the edges located by this method have better detail than the baseline model. In the third column, the model of this invention does not locate any unnecessary details.

[0097] Figure 3 A visual comparison chart of the method of this invention and the TruFor model (a previous method) is shown, which demonstrates that the method can effectively solve the problems of false detection and false negatives.

[0098] like Figure 4 As shown in the figure, a visual comparison of the method of the present invention and existing methods is presented. The first row is the tampered image, the second row is the mask (white areas represent the tampered areas), the third row is the visual localization map of the present invention, and the last four rows are visualization maps of other models. It can be seen from the figure that the method of the present invention locates the tampered area more accurately, clearly, and completely.

[0099] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalent ones. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image forgery localization and detection method based on collaborative difference optimization and multimodal perception, characterized in that, Includes the following steps: S1. Obtain the RGB image to be detected and perform random data augmentation on the RGB image; S2. The RGB image is processed by a noise extractor to extract noise information, resulting in a noise map; the RGB image and the noise map are then input into a pre-trained cross-modal information encoder (CMX) to obtain multi-scale features; S3. The multi-scale features are processed by an anomaly encoder to obtain an initial prediction map; the anomaly encoder includes a multilayer perceptron (MLP), a linear fusion block, and a linear prediction block; the multi-scale features are processed by the multilayer perceptron (MLP) to obtain multilayer mapping features; S4. Input the RGB image X into the pre-trained ViT model to extract the global features of the image; The global features and multi-layer mapping features are input into the cross-scale feature enhancement module for interactive optimization to obtain a refined localization map. S5. Merge the initial prediction map and the refined localization map to obtain the final localization map; S6. Iteratively optimize the above process using a loss function.

2. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 1, characterized in that, The random data augmentation described in step S1 includes semantically aware random tampering operations, basic image transformation, and imaging process simulation.

3. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 2, characterized in that, Step S2 specifically includes: The S21.RGB image X is processed by a pre-trained Noiseprint++ noise extractor, an Srm noise extractor, and a Bayar noise extractor to obtain the first noise-assisted representation. Second noise auxiliary representation Third noise auxiliary representation The first noise-assisted representation Second noise auxiliary representation and third noise auxiliary representation The hybrid features are obtained after fusion module FM. ; S22. Combine the RGB image X with blending features The first feature is obtained by inputting it into a pre-trained cross-modal information encoder (CMX). Second feature Third feature and the fourth feature The cross-modal information encoder (CMX) employs a cross-attention mechanism to establish RGB image X and mixed features. Channel attention association.

4. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 3, characterized in that, Step S3 specifically includes: The linear fusion block includes a convolutional block, batch normalization, and a ReLU activation function; the linear prediction block includes a convolutional block, a Dropout layer, and an upsampling layer. The first feature Second feature Third feature and the fourth feature After weighted processing by a multilayer perceptron (MLP), multilayer mapping features are obtained. , ; Multi-layer mapping features After being input into a linear fusion block for fusion, fused features are obtained; these fused features are then processed by a linear prediction block to obtain an initial prediction map. .

5. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 4, characterized in that, In step S4, global features are... and multi-layer mapping features The input to the cross-scale feature enhancement module yields the refined localization map Y2. The specific process is as follows: The first mapping feature Second mapping feature Third mapping features Fourth mapping feature Based on their receptive fields, they are divided into local groups ( ) and regional groups ( Then, compare them with the global features respectively. Perform a difference operation, then multiply the difference result element-wise with the corresponding mapped feature to obtain the enhanced feature. , The formula is expressed as follows: ， in, This represents the element-wise multiplication operation; Indicates the difference operation; enhances features. With global features By performing fusion, local group fusion features are obtained. and regional group fusion characteristics The formula is expressed as follows: ， ， in, This indicates splicing along the channel dimension; , , , These represent the first, second, third, and fourth enhancement features, respectively; local group fusion features. and regional group fusion characteristics After processing, the refined positioning map Y2 is obtained. The mask G is used for training and optimization.

6. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 5, characterized in that, In step S5, the initial prediction map is... The final prediction map is obtained by element-wise multiplication with the refined localization map Y2. .

7. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 6, characterized in that, Step S6 includes: For the final prediction map For each pixel in the texture vector, calculate its absolute difference from its neighboring pixels to form a texture vector. , ;in, Indicates the final prediction graph The i-th pixel in the neighborhood of the i-th pixel Texture vector of pixels; Indicates the final prediction graph The i-th pixel in the neighborhood of the i-th pixel A texture vector of 1 pixel, , , This represents the prediction result for a pixel. Indicates the middle pixel. Indicates the middle pixel The j-th pixel in the neighborhood, This represents the k×k pixels in the neighborhood of the i-th pixel; For the first noise auxiliary representation Second noise auxiliary representation and third noise auxiliary representation Texture vectors of the three noise modalities are calculated to obtain the texture vectors of the noise images. The formula is expressed as follows: ， ， in, Represents the neighborhood of the i-th pixel under the m-th noise mode. Texture vector of pixels; Represents an exponential function; Indicates hyperparameters; This represents the texture vector of the i-th pixel under the m-th noise mode; This represents the texture vector of the j-th pixel under the m-th noise mode; Indicates the number of noise modal image types; Represents the square of the Euclidean distance; The joint consistency loss function is expressed as follows: , in, Represents a binary boundary mask. This represents the joint consistency loss function.

8. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 7, characterized in that, The loss functions in step S6 also include the weighted cross-entropy loss function and the Dice loss function.

9. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 8, characterized in that, The total loss function is , ,in, , , These represent the weighted cross-entropy loss function. Weighting coefficients, Dice loss function Weighting coefficients, joint consensus loss function The weighting coefficients.

10. The image forgery localization and detection method based on collaborative difference optimization and multimodal perception according to claim 9, characterized in that, The binary boundary mask The acquisition process is as follows: the final prediction map Convert to binary tags Distinguish between foreground and background to obtain a binary matrix L; extract the boundary mask from the binary matrix L using the erosion-dilation technique to obtain the binary boundary mask. .

Images