A robust tampering localization method for remote sensing images based on segmentation-all-model (SAM)

Abstract

The present invention provides a remote sensing image robust tampering localization method based on a segmentation-all-model (SAM), comprising the following steps: step 1, obtaining an input remote sensing image I and a noise image R; step 2, inputting the remote sensing image I and the noise image R into an optical flow SAM encoder and a noise flow SAM encoder, respectively, which form a dual-stream SAM encoder with shared parameters; step 3, dynamically calibrating multi-level semantic features through a dynamic tampering signal enhancement module (DTSEM); step 4, inputting the calibrated features into a parallel branch forgery decoder (PFD); step 5, optimizing a loss function; and step 6, performing inference phase processing. The present invention utilizes the high sensitivity of SAM to feature distribution differences to effectively capture tampering traces weakened by post-processing and improve localization accuracy.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a remote sensing image robust tampering positioning method based on a Segmentation Everything Model (SAM). Background Art

[0002] With the rapid development of remote sensing technology, high-resolution remote sensing imagery has been widely used in fields such as land and resources surveys, urban planning, and environmental monitoring. However, with the widespread adoption of image editing techniques, remote sensing image tampering is on the rise, seriously threatening the reliability and security of remote sensing data. Therefore, developing efficient and accurate remote sensing image tampering detection technology is of great practical significance.

[0003] Currently, remote sensing image tampering detection methods are primarily divided into traditional feature-based methods and deep learning-based methods. Traditional methods rely primarily on handcrafted features, such as noise residuals and JPEG compression artifacts, but these methods perform poorly when faced with complex scenes and diverse tampering types. With the advancement of deep learning technology, tampering detection methods based on convolutional neural networks (CNNs) and Transformers have gradually become a research hotspot.

[0004] In the field of remote sensing image processing, multi-scale feature extraction and fusion are key technologies for improving detection accuracy. CN117809198A discloses a remote sensing image saliency detection method based on a multi-scale feature aggregation network. This method uses dilated convolutions with different dilation rates and feature attention guidance to fuse feature information of different scales from the backbone network branches. The feature alignment module uses deformable convolutions for feature alignment and aggregation, enhancing the model's ability to perceive objects of different scales. This multi-scale feature fusion strategy provides a useful reference for remote sensing image tampering detection.

[0005] The dual-branch network structure also shows good performance in remote sensing image processing. CN118314353B proposes a remote sensing image segmentation method based on dual-branch multi-scale feature fusion, using two parallel branches of CNN and Transformer to extract local features and global features at different resolutions respectively, realizing the effective integration of global-local information. Similarly, CN119540558A introduces a Transformer remote sensing semantic segmentation method based on spatial channel cross decoding, which decodes spatial features and channel features through a dual-stream decoder network, and uses a deformable attention mechanism to capture the contextual dependencies of spatial and channel features, thereby enhancing the feature representation capability.

[0006] In the field of image tampering detection, CN117893858A discloses a method for localizing image tampering that integrates multi-level, multi-scale, and boundary information. This method uses a pyramid vision Transformer backbone network to extract multi-level tampering features, strengthens feature representation through a multi-scale tampering feature enhancement module, and uses a tampering boundary information module to specifically model the boundary information of the tampered area. This method of integrating multi-level features and boundary information is of great significance for improving the accuracy of tampering detection.

[0007] Furthermore, CN117612029B proposes a remote sensing image target detection method based on progressive feature smoothing and scale-adaptive dilated convolution. By constructing an adaptive feature extraction network, this method effectively reduces the missed detection rate in remote sensing images and improves target detection accuracy. This scale-adaptive design approach also has important implications for remote sensing image tampering detection.

[0008] However, existing remote sensing image tampering detection methods still have the following problems:

[0009] First, existing methods have poor perception of subtle differences in feature distribution, especially after images undergo post-processing operations (such as Gaussian blur and JPEG compression), which weaken the original tampering traces and lead to a significant decrease in detection performance. Existing methods lack robustness to such post-processing interference and have difficulty detecting weakened tampering traces.

[0010] Secondly, remote sensing image textures are complex and diverse, containing a large number of natural textures and artificial structures. Existing methods are easily disturbed by these complex textures, making it difficult to effectively distinguish normal texture changes from tampering traces, resulting in a high false positive rate. This texture interference problem is particularly prominent in high-resolution remote sensing images.

[0011] Third, existing methods lack explicit mechanisms for identifying tampering traces and rely primarily on the model's inherent feature extraction capabilities. When faced with post-processed tampered images, they tend to converge to suboptimal solutions, making it difficult to effectively explore and extract obscured tampering traces. This lack of targeted feature extraction limits the model's detection performance in complex scenarios.

[0012] Therefore, it is urgent to develop a remote sensing image tampering detection method that can effectively deal with post-processing interference, complex texture background, and has the ability to guide explicit tampering traces, so as to improve the accuracy and robustness of detection. Summary of the Invention

[0013] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a robust tampering localization method for remote sensing images based on the Segment Anything Model (SAM), comprising the following steps:

[0014] Step 1, feature input and preprocessing: obtain input remote sensing image , where H and W represent the height and width of the image respectively, and the number of channels of the image is 3; the corresponding noise image is generated by the rich steganalysis model SRM (Steganalysis Rich Model) filter , construct a dual-stream input containing optical information and high-frequency noise information, where represents the real number space;

[0015] Step 2, dual-stream feature encoding: The remote sensing image I and the noise image R are input into the optical stream SAM encoder and the noise stream SAM encoder respectively. The optical stream SAM encoder and the noise stream SAM encoder form a dual-stream SAM encoder with shared parameters.

[0016] The optical flow SAM encoder and the noise flow SAM encoder have the same structure, both of which contain 4 modules, and the module number is represented by i. All four modules adopt the same structural design. Each module consists of a cascade of an adaptive multi-scale feature adapter (AMFA) and a Transformer layer. The AMFA processes input features using two or more sets of convolutional kernels of different scales to capture feature information at different scales. The Transformer layer uses a self-attention mechanism to globally model features and enhance feature expression capabilities.

[0017] The optical flow SAM encoder and the noise flow SAM encoder output multi-level features respectively and ,in It is the multi-level semantic feature output by the optical flow (i.e., the input is the remote sensing image I) after passing through the i-th module in the optical flow SAM encoder. Optical represents optics, corresponding to the semantic features extracted from the remote sensing image input; It is the multi-level tampered noise feature output by the noise stream (i.e., the input is the noise image R) after passing through the i-th module in the noise stream SAM encoder, where noise represents noise, corresponding to the tampered noise feature extracted from the noise image input;

[0018] Step 3: Dynamic tampering signal enhancement: Through the dynamic tampering signal enhancement module DTSEM, the multi-level tampering noise characteristics are used Multi-level semantic features Perform dynamic calibration and calculate feature weights through a gating mechanism , output calibration characteristics , where Conv represents the convolution operation, Convolution features of noise streams are used to enhance effective tampering signals and suppress texture interference;

[0019] Step 4, parallel branch forged decoding: the calibration feature Input to a parallel branch forged decoder PFD; the parallel branch forged decoder PFD includes four parallel branches, namely a first branch, a second branch, a third branch and a fourth branch;

[0020] The first branch includes 1×1 convolution, regular convolution layer, normalization layer, activation function layer and calculation layer for deep supervision;

[0021] The second branch includes a 3×3 dilated convolution with a dilation rate of 6, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision;

[0022] The third branch includes a 3×3 dilated convolution with a dilation rate of 12, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision;

[0023] The fourth branch includes a 3×3 dilated convolution with a dilation rate of 18, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision;

[0024] The multi-scale tampering features are extracted through convolution branches with different expansion rates, and the pixel-level classification of each layer output in the parallel branch forgery decoder PFD is combined with the deep supervision mechanism, and finally the tampering area mask that integrates multi-scale information is output. ; Based on steps 1 to 4, the construction of the dual-stream SAM model was completed;

[0025] Step 5, loss function optimization: In order to effectively improve the performance of the model in detecting tampered areas in remote sensing images, a combination of weighted intersection-over-union loss (wIoU) and weighted binary cross entropy loss (wBCE) is adopted.

[0026] Step 6: The dual-stream SAM model performs inference processing: In the inference stage, the dual-stream SAM model supports dynamic input size. For input remote sensing images of different resolutions, the size of the remote sensing images will be uniformly adjusted by interpolation preprocessing. For the images of the Post-FakeV dataset, the size will be uniformly adjusted to Size; for images in the Post-FakeL dataset, unified to Size; The two-stream SAM model performs inference calculation based on the adjusted image, and after obtaining the result, restores the output to the original size of the image through interpolation operation.

[0027] In step 1, a 3×3 SRM filter bank is applied to the remote sensing image I to extract the residual signal containing the high-frequency noise pattern introduced by the tampering. The formula is:

[0028] ,

[0029] The SRM filter bank contains edge detection kernels in horizontal, vertical and diagonal directions, which is used to explicitly model the noise inconsistencies in the image acquisition and tampering process.

[0030] In step 2, the adaptive multi-scale feature adapter AMFA includes a multi-scale feature extraction unit and a hierarchical interactive fusion unit;

[0031] The adaptive multi-scale feature adapter AMFA performs the following operations:

[0032] Input features First, the linear projection layer is used to reduce the dimension, and the low-dimensional features are obtained through GeLU activation and reshaping operations. , the formula is:

[0033] ,

[0034] in is the dimension reduction matrix, RS is the reshaping operation;

[0035] Subsequently, the features are subjected to three 3×3 dilated convolutions with different dilation rates of 6, 12, and 18 to extract multi-scale information, and global context features are obtained through global average pooling, which is expressed as:

[0036] ,

[0037] in represents 1×1 convolution, It is a dilated convolution with expansion rates of 6, 12, and 18, corresponding to ; AAP is the adaptive average pooling layer, and Up is the bilinear interpolation upsampling; finally, the multi-scale features are concatenated with the original feature channels to form a feature set containing multi-scale information;

[0038] The hierarchical interactive fusion unit performs cross-layer fusion of multi-scale features, and the formula is:

[0039] ,

[0040] in, represents the jth feature at the lth level after hierarchical interaction fusion; is the jth original multi-scale feature at the lth level; is the activation function, used to introduce nonlinearity; Represents a 3×3 convolution operation;

[0041] The feature aggregation and recovery unit integrates the multi-scale features after hierarchical interaction and restores the dimensions. The formula is:

[0042] ,

[0043] ,

[0044] in, It means that after a series of feature processing operations (including multi-scale feature extraction, hierarchical interactive fusion, etc.), the aggregated features obtained at the lth level are the overall feature representation after comprehensive processing of multiple related features at this level. It is a linear projection layer that restores the feature dimension to the original input dimension C. It is the final output feature obtained after multi-step feature processing and dimension restoration at the lth level.

[0045] In step 2, the Transformer layer adopts a hierarchical feature extraction architecture. In the optical stream SAM encoder and the noise stream SAM encoder, the first and second modules are low-level modules, and the third and fourth modules are high-level modules.

[0046] The low-level modules focus on local texture differences, the high-level modules capture the global semantic distribution, and the representation migration from semantic features to tampering features is achieved through the adaptive multi-scale feature adapter AMFA.

[0047] In step 3, the gating mechanism includes:

[0048] Multi-level tampering noise features Perform convolution, batch normalization, and PReLU activation (using parameterized rectified linear unit PReLU) operations to generate transition features ;

[0049] Establish a two-layer feature transformation network with the same structure, and perform convolution, batch normalization, and PReLU activation operations respectively;

[0050] Transition Features With multi-level semantic features After channel splicing, the two-layer feature transformation network is input, and after two rounds of convolution, batch normalization, PReLU activation operations, and then global average pooling GAP (Global Average Pooling) operations, the gate value is generated. The global average pooling operation performs an average calculation on the feature map in the spatial dimension to obtain a feature vector of fixed length. This process enables adaptive selection of tampering-related noise features.

[0051] In step 4, the first branch, second branch, third branch and fourth branch are used to capture local detail features, medium-scale features, larger-scale features and global context features, respectively, and through jump connections, the shallow feature map that retains high-resolution details and the deep feature map containing abstract semantic information are fused by channel concatenation or weighted summation.

[0052] In step 5, the weighting factor in the loss function is designed based on the local neighborhood mean. :

[0053] ,

[0054] in is located at image coordinates The weighting factor of the pixel in the loss function is used to adjust the weight of the pixel in the loss calculation; True mask Center coordinates Pixel Neighborhood mean; is the coordinate in the real mask G The pixel value at .

[0055] In step 6, the dual-stream SAM model will first adjust the size of the remote sensing images uniformly by interpolation preprocessing for the input remote sensing images of different resolutions. For the images of the Post-FakeV dataset, the size is unified to Size; for images in the Post-FakeL dataset, unified to Size; The two-stream SAM model performs inference calculation based on the adjusted image, and after obtaining the result, restores the output to the original size of the image through interpolation operation.

[0056] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the method.

[0057] The present invention also provides a storage medium storing a computer program or instruction, which executes the steps of the method when the computer program or instruction is run on a computer.

[0058] The present invention has the following beneficial effects: High sensitivity of SAM: By utilizing the high sensitivity of SAM to feature distribution differences, it is possible to effectively capture tampering traces weakened by post-processing and improve positioning accuracy.

[0059] AMFA's domain adaptation capability: bridges the gap between natural images and remote sensing images, semantic and non-semantic features, and enhances the pertinence of feature extraction.

[0060] DTSEM's anti-interference capability: Dynamically suppresses texture interference based on a gating mechanism, explicitly enhances tampering signals, and reduces false alarm rates.

[0061] Multi-scale modeling of PFD: Combined with deep supervision, multi-scale features are extracted to improve the detection ability of tampered areas of different scales.

[0062] Post-processing robustness verification: Experiments on the Post-FakeV / Post-FakeL datasets show that the proposed method is significantly superior to existing methods and has stronger generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a flow chart of the method of the present invention, showing the overall process from creating a data set to the final verification of the advantages of the method.

[0064] Figure 2 This is an architectural diagram of the method described in the present invention, showing the interconnection relationship between the dual-stream SAM encoder, AMFA, DTSEM and PFD, as well as the processing paths of the input image, noise image and mask.

[0065] Figure 3 This is a structural diagram of the Adaptive Multi-Scale Feature Adapter (AMFA), which details its internal process of multi-scale feature processing through operations such as convolution and pooling with different expansion rates.

[0066] Figure 4 This is a schematic diagram of the structure of the dynamic tampering signal enhancement module (DTSEM), which depicts the gating mechanism and feature fusion process for dynamically calibrating optical flow features using noise flow features.

[0067] Figure 5 This is a structural diagram of the parallel branch forgery decoder (PFD), which shows its process of extracting multi-scale tampering features through multi-branch convolution operations and processing them in combination with a deep supervision mechanism. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0069] like Figure 1 As shown, in a first embodiment of the present invention, a remote sensing image robust tampering location method based on a segmentation everything model (SAM) is provided, comprising the following steps:

[0070] The steps include:

[0071] Step 1, feature input and preprocessing: obtain input remote sensing image , where H and W represent the height and width of the image respectively, and the number of channels of the image is 3; the corresponding noise image is generated by the rich steganalysis model SRM (Steganalysis Rich Model) filter , construct a dual-stream input containing optical information and high-frequency noise information, where represents the real number space;

[0072] Step 2, dual-stream feature encoding: Figure 2 As shown, the remote sensing image I and the noise image R are input into the optical stream SAM encoder and the noise stream SAM encoder respectively, and the optical stream SAM encoder and the noise stream SAM encoder form a dual-stream SAM encoder with shared parameters;

[0073] The optical flow SAM encoder and the noise flow SAM encoder have the same structure, both of which contain 4 modules, and the module number is represented by i. , the four modules all adopt the same structural design, each module is composed of an adaptive multi-scale feature adapter (AMFA) and a cascade of Transformer layers, among which the adaptive multi-scale feature adapter AMFA processes the input features through two or more sets of convolution kernels of different scales to capture feature information at different scales; the Transformer layer globally models the features based on the self-attention mechanism to enhance the feature expression capability; although the four modules have the same structure, at different levels, they will learn and adjust their respective parameters according to the differences in the resolution and semantic level of the feature maps they process. As the level i increases, the resolution of the feature map gradually decreases, while the semantic information gradually increases, thereby realizing the gradual extraction and conversion from low-level detail features to high-level semantic features. In this way, the optical flow SAM encoder and the noise stream SAM encoder output multi-level features respectively. and ,in It is the multi-level semantic feature output by the optical flow (i.e., the input is the remote sensing image I) after passing through the i-th module in the optical flow SAM encoder. Optical represents optics, corresponding to the semantic features extracted from the remote sensing image input; It is the multi-level tampered noise feature output by the noise stream (i.e., the input is the noise image R) after passing through the i-th module in the noise stream SAM encoder. The noise represents noise, which corresponds to the tampered noise feature extracted from the noise image input, realizing the parallel modeling of semantic features and tampered noise features.

[0074] Step 3, dynamic tampering signal enhancement: Figure 4 As shown, the dynamic tampering signal enhancement module (DTSEM) uses multi-level tampering noise features Multi-level semantic features Perform dynamic calibration and calculate feature weights through a gating mechanism , output calibration characteristics , where Conv represents the convolution operation, Convolution features of noise streams are used to enhance effective tampering signals and suppress texture interference;

[0075] Step 4, parallel branch forged decoding: Figure 5 As shown, the calibration characteristics Input to a parallel branch forged decoder PFD; the parallel branch forged decoder PFD includes four parallel branches, namely a first branch, a second branch, a third branch and a fourth branch;

[0076] The first branch includes 1×1 convolution, regular convolution layer, normalization layer, activation function layer and calculation layer for deep supervision;

[0077] The second branch includes a 3×3 dilated convolution with a dilation rate of 6, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision;

[0078] The third branch includes a 3×3 dilated convolution with a dilation rate of 12, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision;

[0079] The fourth branch includes a 3×3 dilated convolution with a dilation rate of 18, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision;

[0080] The multi-scale tampering features are extracted through convolution branches with different expansion rates, and the pixel-level classification of each layer output in the parallel branch forgery decoder PFD is combined with the deep supervision mechanism, and finally the tampering area mask that integrates multi-scale information is output. ; Based on steps 1 to 4, the construction of the dual-stream SAM model was completed;

[0081] Step 5, loss function optimization: In order to effectively improve the performance of the model in detecting tampered areas in remote sensing images, a combination of weighted intersection-over-union loss (wIoU) and weighted binary cross entropy loss (wBCE) is adopted.

[0082] The total loss function comprehensively considers the prediction performance of the model at different levels. , the feature map of its output Reflects the prediction result of this layer on the tampered area in the image.

[0083] When designing the loss function, the weighting factor is introduced , for the weighted binary cross entropy loss function The weighting factor adjusts the weight of pixels in the loss calculation to alleviate the class imbalance problem and make the model pay more attention to small tampered areas. It is calculated based on information such as the mean of the pixel neighborhood in the ground truth mask G, and measures the degree of difference in classification between the pixel-level probability predicted by the model and the ground truth mask.

[0084] Weighted Intersection-over-Union loss function The weighting factor is used to measure the overlap difference between the model prediction area and the real tampered area. Especially when the tampered area is small and the real area accounts for a large proportion, it can enhance the model's attention to different areas and improve the ability to identify small-scale tampered areas. As a standard reference that accurately represents the actual tampered area in the image, it provides a reliable basis for loss calculation.

[0085] Furthermore, to enhance the model's ability to capture details and identify forgeries at varying scales, a deep supervision mechanism is introduced to the outputs of the first three layers of the PFD. Through this hierarchical supervision, the model receives effective feedback at different levels, improving its ability to perceive complex tampering patterns and ultimately enabling more accurate detection of tampered areas in remote sensing images.

[0086] Step 6, reasoning stage processing: In the reasoning stage, the two-stream SAM model supports dynamic input size. For input remote sensing images of different resolutions, the size of the remote sensing images will be uniformly adjusted by interpolation preprocessing. For the images of the Post-FakeV dataset, the size will be uniformly adjusted to Size; for images in the Post-FakeL dataset, unified to size; the model performs inference calculations based on the adjusted image, and after obtaining the result, restores the output to the original size of the image through interpolation; this processing method can ensure that the model has the ability to accurately locate tampered areas in high-resolution remote sensing images, effectively cope with the diverse resolutions of remote sensing images in actual applications, and improve the practicality and effectiveness of the model in different scenarios.

[0087] In step 1, the input remote sensing image I can be a satellite or aerial remote sensing image of various resolutions, including but not limited to visible light band, infrared band or multispectral remote sensing image. In this embodiment, the resolution of the input image can be 352×352 pixels (for Post-FakeV dataset) or 512×512 pixels (for Post-FakeL dataset) according to the dataset type. Apply a 3×3 SRM filter bank to extract the residual signal containing the high-frequency noise pattern introduced by the tampering, as follows:

[0088] ,

[0089] The SRM filter bank contains edge detection kernels in horizontal, vertical and diagonal directions, which is used to explicitly model the noise inconsistencies in the image acquisition and tampering process.

[0090] In step 2, if Figure 3 As shown, the adaptive multi-scale feature adapter (AMFA) includes a multi-scale feature extraction unit and a hierarchical interactive fusion unit;

[0091] The multi-scale feature extraction unit extracts multi-scale features containing local details and global context through 1×1 convolution, dilation rate 6 / 12 / 18 hole convolution and global average pooling. First, the linear projection layer is used to reduce the dimension, and the low-dimensional features are obtained through GeLU activation and reshaping operations. , the formula is:

[0092] ,

[0093] in is a reduced dimensionality matrix (reduction factor r=4), and RS is a reshaping operation;

[0094] Subsequently, the features are subjected to three 3×3 dilated convolutions with different dilation rates of 6, 12, and 18 to extract multi-scale information, and global context features are obtained through global average pooling, which is expressed as:

[0095] ,

[0096] in represents 1×1 convolution, RConv represents hole convolution, Represents dilation rates of 6, 12, and 18, corresponding to the dilation rates of 6, 12, and 18, respectively. ; AAP is the adaptive average pooling layer, and Up is the bilinear interpolation upsampling; finally, the multi-scale features are concatenated with the original feature channels to form a feature set containing multi-scale information;

[0097] The hierarchical interactive fusion unit performs cross-layer fusion of multi-scale features, and the formula is:

[0098] ,

[0099] in, Indicates the first level under the first level after hierarchical interaction fusion Features For the first level Original multi-scale features; is the activation function, used to introduce nonlinearity; Represents a 3×3 convolution operation. When , the fusion feature is the original multi-scale feature; when When the fusion feature is The j-1th and j-1th original multi-scale features are first added together, and then processed by 3×3 convolution and ReLU activation to achieve hierarchical interaction of features of different scales and enhance sensitivity to subtle tampering traces.

[0100] The feature aggregation and recovery unit integrates the multi-scale features after hierarchical interaction and restores the dimensions. The formula is:

[0101] ,

[0102] ,

[0103] in, It is a linear projection layer that restores the feature dimension to the original input dimension Finally, the processed features are added to the original input features through residual connections, which not only preserves the multi-scale interaction information but also avoids feature representation degradation.

[0104] In step 2, the Transformer layer adopts a hierarchical feature extraction architecture. In the optical stream SAM encoder and the noise stream SAM encoder, the first and second modules are low-level modules, and the third and fourth modules are high-level modules.

[0105] Low-level modules (such as ) focuses on pixel-level texture differences through small-scale convolution kernels and local attention mechanisms to capture subtle structural anomalies in the tampered area; high-level modules (such as ) With the help of large-scale dilated convolution and global self-attention mechanism, scene-level semantic distribution is modeled to identify global semantic inconsistencies caused by tampering. The processing flow of each module is as follows: the upper layer output features are first fused by AMFA at multiple scales (including 1×1 convolution, dilated convolution with expansion rates of 6, 12, and 18, and global average pooling), and then cross-position dependency modeling is achieved through the Transformer layer, and finally the current layer feature representation is output. Among them, the first layer module directly uses the original image features (optical flow or noise flow ) as input, and construct a feature pyramid from low-level details to high-level semantics layer by layer, providing multi-granularity feature support for subsequent tampering localization.

[0106] In step 3, the gating mechanism includes:

[0107] Multi-level tampering noise features Perform convolution, batch normalization, and activation (using parameterized rectified linear unit PReLU) operations to generate transition features This series of operations is referred to as convolution, batch normalization, and parameterized rectified linear unit activation (CBPR). Specifically, the convolution operation uses the convolution kernel to tamper with the noise features of multiple layers. Extract local feature information; batch normalization normalizes the convolutional features in the batch dimension to make the data distribution more stable, which helps accelerate model training and alleviate gradient problems; the parametric rectified linear unit (PReLU) in the activation operation is used as an activation function. It introduces learnable parameters to adaptively adjust the function slope, adding nonlinear factors to the model and improving its expressiveness.

[0108] A two-layer feature transformation (CBPR) network is established. The two layers of feature transformation network have the same structure and perform convolution, batch normalization, and PReLU activation operations respectively.

[0109] Transition Features With multi-level semantic features After channel splicing, the two-layer feature transformation network is input, and after two rounds of convolution, batch normalization, PReLU activation operations, and then global average pooling GAP (Global Average Pooling) operations, the gate value is generated. The global average pooling operation performs an average calculation on the feature map in the spatial dimension to obtain a feature vector of fixed length. This process enables adaptive selection of tampering-related noise features.

[0110] In step 4, the first branch, the second branch, the third branch, and the fourth branch are respectively used to capture local detail features (used to detect small-scale tampering, such as pixel-level forgery, slight erasure, etc., the area of such tampered areas usually accounts for less than 5% of the total image area), medium-scale features (capable of detecting edge tampering, such as image splicing traces and texture anomalies, the area of the tampered area accounts for 5% - 20% of the total image area), and large-scale features (suitable for locating medium-sized tampered areas, such as area replacement, content copying and moving, the area of the tampered area accounts for approximately 20% - 20% of the total image area). 50%) and global context features (capable of identifying large-scale semantic inconsistencies, such as whole-region synthesis and scene tampering, where the area of the tampered region accounts for more than 50% of the total image area), and through jump connections, shallow feature maps that retain high-resolution details (such as edges and textures, from the early layers of the encoder or the low-level branches of PFD) are fused with deep feature maps containing abstract semantic information (such as objects and scenes, from the high-level branches of PFD) by channel concatenation or weighted summation, thereby improving the positioning accuracy of complex tampering patterns and enabling PFD to simultaneously focus on multi-scale tampering features from the pixel level to the scene level, effectively dealing with diverse forgery methods in remote sensing images.

[0111] In step 5, the weighted binary cross entropy loss (wBCE) aims to optimize pixel-level classification accuracy and alleviate the class imbalance problem. Binary cross entropy loss (BCE) is commonly used in pixel-level classification tasks to measure the difference between the predicted probability map and the true mask. The formula is:

[0112] ,

[0113] in, Represents the binary cross entropy loss. S represents the prediction result of the model, and the superscript i represents the different levels of the parallel branch forged decoder (PFD). That is Hierarchical prediction results. G is a mask. In order to highlight key areas and alleviate category imbalance, a weighting factor is innovatively introduced. , we get the weighted binary cross entropy loss :

[0114] ,

[0115] The weighting factor is calculated using the following formula: :

[0116] ,

[0117] in is located at image coordinates The weighting factor of the pixel in the loss function is used to adjust the weight of the pixel in the loss calculation to alleviate the class imbalance problem and enhance the attention to the tampered area of small objects; is the coordinate in the feature map output by the PFD layer i The pixel value at reflects the level of the coordinate in the image The prediction result of whether the area is tampered. True mask Center coordinates Pixel Neighborhood mean, that is, calculating the mean value centered on the pixel The average value of pixel values in the large and small neighborhoods; is the coordinate in the real mask G The pixel value at that location indicates whether it belongs to the tampered area. By calculating the weighting factor in this way, we can reasonably allocate weights when calculating the loss, making the model pay more attention to the tampered area of small objects, and effectively alleviating the impact of class imbalance.

[0118] The Intersection over Union (IoU) is a key metric for measuring the degree of overlap between the predicted region and the masked region. The weighted Intersection over Union (wIoU) loss helps to enhance the model's attention to different regions, especially when the tampered region is small and the true region occupies a large part of the image. This improves the model's ability to identify small-scale tampered regions:

[0119] ,

[0120] ,

[0121] ,

[0122] Here i represents the different levels of the parallel branch forgery decoder (PFD), For all pixel positions, according to the weighting factor , model prediction value and the true mask value An intermediate quantity is calculated to measure the intersection of the predicted result and the true mask under weighted considerations. The same is true for all pixel positions, according to the weighting factor , model prediction value and the true mask value The calculated intermediate quantity measures the union of the predicted result and the true mask taking into account the weighting. Indicates the prediction result at the i-th level of the parallel branch forgery decoder (PFD) With the real mask Compute weighted intersection-over-union loss.

[0123] Overall loss function yes and The combination of is expressed as:

[0124] ,

[0125] In addition, to enhance the model's ability to capture details and identify forgeries of different scales, a deep supervision mechanism is introduced for the segmentation output of each layer. Through this layer-by-layer guidance, the model obtains effective feedback at different levels, thereby improving its ability to perceive complex forgery patterns. The final total loss function is Defined as:

[0126] .

[0127] In step 6, the SAM model's inference phase supports dynamic input size, achieving compatibility with remote sensing images of varying resolutions through interpolation preprocessing. This process uses methods such as bilinear interpolation to scale the image, minimizing information loss while maintaining feature integrity. This ensures the model can effectively extract tampering features at all scales, improving its ability to capture subtle tampering traces in high-resolution remote sensing images and enhancing the model's generalization performance in complex real-world scenarios.

[0128] Training and Inference Strategy: In one embodiment of the present invention, a progressive enhancement strategy is employed during training: pre-training is performed on the original, unprocessed dataset, followed by fine-tuning on the Post-FakeV / Post-FakeL datasets to improve the model's robustness to post-processing perturbations. Specifically, the pre-training phase uses a dataset of tampered original remote sensing images to learn basic representations of tampering features. The fine-tuning phase utilizes the constructed post-processed datasets, Post-FakeV and Post-FakeL, which contain both single and combined post-processing operations. Single post-processing operations include Gaussian blur (radius 0.1, 0.3, 0.5, 1), Gaussian noise (variance 3, 9, 15, 20), and JPEG compression (quality 30, 50, 70, 90); combined post-processing operations include (blur radius 0.3) + (noise variance 9) + (JPEG quality 70), simulating the multiple perturbations encountered in real-world attack scenarios.

[0129] During the model's inference phase, dynamic input resizing is supported. Remote sensing images of varying resolutions are unified to 352×352 (Post-FakeV) or 512×512 (Post-FakeL) through interpolation preprocessing, and restored to their original size upon output, ensuring the ability to locate tampering in high-resolution remote sensing images. Specifically, for input remote sensing images of any size, bilinear interpolation is first used to resize them to a standard size. The model then processes them to predict tampering masks, which are then interpolated back to their original size to maintain consistency with the input image.

[0130] Experimental results: The method of this embodiment demonstrates excellent tampering localization performance under various post-processing interference conditions. For example, under Gaussian blur (radius 0.5), the method achieves an F1 score of 0.712, an 8.5% improvement over the state-of-the-art method. Under JPEG compression (quality 50), the method achieves an IoU of 0.658, a 7.2% improvement over the state-of-the-art method. Under Gaussian noise (variance 15), the method achieves a precision of 0.735, a 9.1% improvement over the state-of-the-art method. In particular, under the combined post-processing conditions (blur radius 0.3 + noise variance 9 + JPEG quality 70), the method achieves an F1 score of 0.679, significantly outperforming the state-of-the-art method (0.598), demonstrating its robustness in complex interference environments.

[0131] Ablation experiments analyzed the contribution of each component, showing that removing the AMFA module reduced the F1 score by 4.3%, removing the DTSEM module by 5.7%, removing the noise stream by 7.2%, and replacing the multi-branch decoder (PFD) with a single-branch decoder by 3.8%. These results demonstrate the significant contribution of each component to model performance, particularly the key role of the noise stream and DTSEM modules in improving model robustness.

[0132] The present invention also provides a second embodiment. Based on the first embodiment, this embodiment optimizes the implementation of the adaptive multi-scale feature adapter (AMFA). AMFA includes a multi-scale feature extraction unit and a hierarchical interactive fusion unit. The multi-scale feature extraction unit extracts multi-scale features containing local details and global context through 1×1 convolution, dilated convolution with expansion rates of 6, 12, and 18, and global average pooling. , where l represents the layer, and the values of l are 1, 2, 3, and 4; m is the index used to distinguish different features. .

[0133] In this embodiment, the multi-scale feature extraction unit is specifically implemented as follows: First, input feature Reduce the channel dimension to 128 through 1×1 convolution to obtain the feature ;Then, After three 3×3 dilated convolutions with different expansion rates (expansion rates are 6, 12, and 18 respectively), the features are obtained. 、 、 At the same time, After global average pooling, 1×1 convolution is performed to obtain global context features. Finally, 、 、 、 、 Splicing in the channel dimension to form a multi-scale feature set.

[0134] The hierarchical interactive fusion unit performs cross-layer fusion of multi-scale features. The formula is: ,in is the weight coefficient of adaptive learning, which is calculated by the attention mechanism. Specifically, the multi-scale features are first After splicing, the weight coefficient is generated by 1×1 convolution and Softmax function , and then perform weighted summation on each scale feature to obtain the final output feature This hierarchical interactive fusion mechanism can adaptively select and combine features of different scales, enhancing sensitivity to subtle tampering traces.

[0135] Experimental results show that the optimized AMFA module can more effectively extract and fuse multi-scale features, improving the F1 score by 1.2% on the Post-FakeV dataset and the IoU by 1.5% on the Post-FakeL dataset, demonstrating its important contribution to improving model performance.

[0136] The present invention also provides a third embodiment. Based on the first embodiment, this embodiment optimizes the gating mechanism of the dynamic tampering signal enhancement module (DTSEM). The gating mechanism of DTSEM includes the gating mechanism of the noise flow feature. Perform convolution-batch normalization-activation (CBPR) operations to generate transition features ;Will Optical flow characteristics After channel splicing, the gated value is generated through a two-layer CBPR network and global average pooling (GAP) , to achieve adaptive selection of tampering-related noise features.

[0137] In this embodiment, the specific implementation of the CBPR operation is as follows: First, a 3×3 convolution is applied to the input features, and the number of channels is adjusted to half of the number of input channels; then, the convolution results are batch normalized to stabilize the training process; finally, the ReLU activation function is applied to introduce nonlinearity. , after a CBPR operation, the transition feature is obtained .

[0138] Gate value The calculation process is optimized as follows: First, and The channels are concatenated in the channel dimension; the number of channels is halved by the first layer of CBPR, and the second layer of CBPR further reduces the number of channels to 64; global average pooling is then applied to obtain channel-level feature vectors; finally, a 1×1 convolution and a Sigmoid activation function are used to generate a gating value in the range [0, 1]. .

[0139] Calibration characteristics The calculation formula is: ,in The noise flow feature is processed by 3×3 convolution. This gating mechanism can adaptively adjust the weights of noise and optical features according to the content of the input features, enhancing the focus on the tampered area and suppressing background interference.

[0140] Experimental results show that the optimized DTSEM module can more accurately identify and enhance tampering-related noise features, improving the F1 score by 2.3% under Gaussian noise (variance 20) and the IoU by 2.7% under JPEG compression (quality 30), demonstrating its important contribution to improving the robustness of the model.

[0141] The present invention also provides a fourth embodiment. Based on the first embodiment, this embodiment optimizes the implementation of the parallel branch forgery decoder (PFD). The PFD adopts a parallel branch structure, each branch contains a hole convolution with a different expansion rate to capture (Expansion rate = 1), (Expansion rate = 6), (Expansion rate = 12), (dilation rate = 18) scale tampering features and fuse shallow details with deep semantic information through skip connections.

[0142] In this embodiment, the specific implementation of PFD is as follows: First, the calibration feature The input is respectively fed into four parallel branches; each branch first adjusts the number of channels to 128 through 1×1 convolution, and then extracts features of a specific scale through 3×3 dilation convolution with corresponding expansion rates (1, 6, 12, 18); then, the features of the four branches are spliced in the channel dimension, and the number of channels is adjusted to 256 through 1×1 convolution to obtain the comprehensive features. .

[0143] In order to fuse shallow details with deep semantic information, PFD adopts a skip connection mechanism: for the lth layer features , and adjust its resolution to the l-1 layer feature through upsampling operation resolution, and then with The layers are concatenated and fused through 1×1 convolution to obtain a new feature representation. This hierarchical fusion starts from the deepest layer and proceeds upward layer by layer, ultimately obtaining a feature representation containing multi-level information.

[0144] During the decoding process, each layer of features will generate an intermediate prediction result after processing , used for deep supervision. The final tampered region mask M is determined by the prediction result of the top layer and restored to the original image resolution through bilinear interpolation.

[0145] Experimental results show that the optimized PFD can more effectively extract and fuse multi-scale tampering features, improving the tampering detection accuracy by 3.5% in complex texture areas and the recall rate by 4.2% in small target tampering areas, demonstrating its important contribution to improving model detection performance.

[0146] The present invention also provides a fifth embodiment. Building on the first embodiment, this embodiment details the construction of the post-processing datasets Post-FakeV and Post-FakeL. These datasets contain both single and combined post-processing operations, and are used to evaluate the robustness of the model under various interference conditions.

[0147] A single post-processing operation includes:

[0148] Gaussian Blur: Blurs the image using Gaussian kernels with radii of 0.1, 0.3, 0.5, and 1, simulating the blur effect of a defocused or compressed image.

[0149] Gaussian Noise: Adds Gaussian noise with variances of 3, 9, 15, and 20 to the image to simulate noise interference during image acquisition or transmission.

[0150] JPEG compression: Compresses images using the JPEG compression algorithm with quality factors of 30, 50, 70, and 90, simulating compression artifacts that may occur during image storage or transmission.

[0151] The comprehensive post-processing operation includes a combination of blur radius 0.3, noise variance 9, and JPEG quality 70, simulating the multiple interferences in real attack scenarios. Specifically, the image is first Gaussian blurred (radius 0.3), then Gaussian noise (variance 9) is added, and finally JPEG compression (quality 70) is performed.

[0152] The dataset construction process is as follows: first, a dataset of original remote sensing image tampering is collected, including real images and their corresponding tampered versions and real masks; then, the same post-processing operation is applied to each pair of images (original image and tampered image) to generate a post-processed version; finally, the post-processed image is paired with the original real mask to form new training and test samples.

[0153] The Post-FakeV dataset is constructed based on medium-resolution remote sensing images, resized to 352 × 352 pixels. The Post-FakeL dataset is constructed based on high-resolution remote sensing images, resized to 512 × 512 pixels. The two datasets contain 5,000 and 3,000 pairs of training samples, respectively, and 1,000 and 600 pairs of testing samples.

[0154] Experimental results show that models trained on these post-processed datasets exhibit greater robustness and are able to effectively cope with various interference conditions. In particular, under the comprehensive post-processing conditions, the F1 score of our method is 12.3% higher than that of the model trained on the original dataset, demonstrating the importance of post-processing datasets in improving model robustness.

[0155] This invention also provides a sixth embodiment. Building on the first embodiment, this embodiment details the implementation of the Transformer layer of a dual-stream SAM encoder. The Transformer layer of the dual-stream SAM encoder uses a hierarchical feature extraction architecture. The bottom-level modules (l = 1, 2) focus on local texture differences, while the top-level modules (l = 3, 4) capture global semantic distribution. AMFA enables representational transfer from semantic features to tampering features.

[0156] In this embodiment, each Transformer layer contains a multi-head self-attention mechanism and a feedforward neural network. Specifically, for the input feature , first through layer normalization processing; then, through the multi-head self-attention mechanism to calculate the attention weights and weighted features; then, the weighted features are added to the input features to form a residual connection; finally, through another layer normalization and feedforward neural network processing, and again apply the residual connection to obtain the output features .

[0157] The calculation process of the multi-head self-attention mechanism is as follows: First, the input feature Linear projection is performed to create three representations: query (Q), key (K), and value (V). The dot product similarity between the query and key is then calculated, and attention weights are obtained through scaling and a softmax function. Finally, the values are weighted and summed using the attention weights to obtain the attention output. To enhance expressiveness, multiple attention heads are computed simultaneously, and their outputs are concatenated and linearly projected to obtain the final output.

[0158] The feedforward neural network consists of two linear layers and a ReLU activation function, which is used to introduce nonlinear transformations and enhance feature expression capabilities. Specifically, the first linear layer expands the feature dimension by 4 times the original dimension. After ReLU activation, the second linear layer restores the dimension to the original dimension.

[0159] In the dual-stream architecture, the optical and noise streams share Transformer layer parameters but perform domain-specific adaptation through the AMFA module. This parameter sharing strategy reduces the number of model parameters while maintaining feature consistency between the two streams, facilitating subsequent feature fusion and calibration.

[0160] Experimental results show that the hierarchical feature extraction architecture can effectively capture tampering features at different levels. The bottom-level module is 8.7% more sensitive to local texture differences than a single architecture, and the high-level module's ability to understand global semantic distribution is 7.5% higher, proving the important contribution of this architecture to improving model performance.

[0161] The present invention also provides a seventh embodiment. Building on the first embodiment, this embodiment details the progressive enhancement strategy employed during training. The progressive enhancement strategy includes pre-training on the original, unprocessed dataset, followed by fine-tuning on the Post-FakeV / Post-FakeL dataset to improve the model's robustness to post-processing interference.

[0162] In this embodiment, the training process is divided into two stages: a pre-training stage and a fine-tuning stage.

[0163] Pre-training: A dataset of original remote sensing image manipulations was used to learn basic representations of manipulation features. Specifically, the AdamW optimizer was used with an initial learning rate of 0.001, a weight decay coefficient of 0.0005, a batch size of 12, and 50 epochs. The learning rate was gradually reduced during training using a cosine annealing strategy.

[0164] Fine-tuning: Using the constructed post-processing datasets Post-FakeV and Post-FakeL, the model was adapted to various interference conditions. Specifically, a small learning rate of 0.0001 was used, while other hyperparameters remained unchanged, and training was performed for 30 epochs. To enhance the model's generalization capabilities, a hybrid training strategy was adopted during fine-tuning, using both original and post-processed data in a 1:2 ratio.

[0165] In addition, data augmentation techniques were used during the training process, including random horizontal flipping, random vertical flipping, random rotation (±15 degrees), random brightness adjustment (±0.2), random contrast adjustment (0.8~1.2), etc., to increase the diversity of training data and improve the generalization ability of the model.

[0166] In order to deal with the problem of class imbalance, in addition to using the weighted loss function, a variant of FocalLoss is also used. By adjusting the weights of difficult and easy samples, the model pays more attention to samples that are difficult to classify. Specifically, the calculation formula of focal loss is: ,in represents the predicted probability, represents the category weight, represents the focus parameter (set to 2 in this embodiment).

[0167] Experimental results demonstrate that the progressive enhancement strategy effectively improves the model's robustness to post-processing interference. Compared to a model trained directly on the post-processed dataset, the model using the progressive enhancement strategy achieves a 3.8% improvement in F1 score under Gaussian blur (radius 1), a 4.2% improvement in IoU under JPEG compression (quality 30), and a 5.1% improvement in precision under Gaussian noise (variance 20), demonstrating the effectiveness of this training strategy.

[0168] The present invention also provides an eighth embodiment. Based on the first embodiment, this embodiment adds the weighting factor in the loss function The calculation method of is described in detail. The weighting factor in the loss function Based on the local neighborhood mean calculation, the formula is: ,in True mask The 31×31 neighborhood mean of the image is used to alleviate the category imbalance problem through a weighted mechanism and enhance the attention to the tampered area of small targets.

[0169] In this embodiment, The calculation process is as follows:

[0170] True mask Apply a 31×31 mean filter to calculate the local neighborhood mean of each pixel position;

[0171] Combine the mean result with the original mask Perform element-wise multiplication to get ;

[0172] Subtract 1 from , and obtain the weighting factor .

[0173] The core idea of this weighting mechanism is:

[0174] For pixels inside the tampered area, most of the pixels in its 31×31 neighborhood are also tampered areas, so The value of is close to 1, corresponding to The value is close to 0;

[0175] For pixels at the edge of the tampered area, their neighborhood contains some non-tampered areas, so The value of is between 0 and 1, corresponding to The value is also between 0 and 1;

[0176] For background pixels far away from the tampered area, The value of is close to 0, corresponding to The value is close to 1.

[0177] In this way, the loss function will give higher weights to the edges of the tampered areas and the tampered areas of small objects, enhancing the model's attention to these areas and improving detection accuracy. Specifically, the calculation formulas for weighted intersection-over-union loss (wIoU) and weighted binary cross entropy loss (wBCE) are:

[0178] ,

[0179] ,

[0180] in, represents the pixel index, and Represent the value of the predicted mask at position i and the value of the real mask at position i, respectively. represents the weighting factor of position i.

[0181] Experimental results show that this weighting mechanism, based on the local neighborhood mean, effectively improves the model's detection performance for small tampered regions and tampered edges. Compared to a loss function using fixed weights, the model using this weighting mechanism achieves a 6.5% improvement in the F1 score for small tampered regions (less than 1% of the image area) and a 5.8% improvement in the Intersection over Union (IoU) for tampered edges, demonstrating the effectiveness of this weighting mechanism.

[0182] The present invention also provides a ninth embodiment. Based on the first embodiment, this embodiment performs a The generation process of the noise image is described in detail. The generation process of includes: applying a 3×3 SRM filter bank to the remote sensing image I to extract the residual signal containing the high-frequency noise pattern introduced by the tampering, the formula is: , where the SRM filter bank contains edge detection kernels in horizontal, vertical and diagonal directions, which is used to explicitly model the noise inconsistency during image acquisition and tampering.

[0183] In this embodiment, the SRM filter bank includes 30 3×3 filters, which are divided into three categories:

[0184] Category 1 (No. 1-10): Detects horizontal edges and texture changes;

[0185] The second category (No. 11-20): detects vertical edges and texture changes;

[0186] The third category (numbers 21-30): detects diagonal edges and texture changes. The weight of each filter is designed based on different direction and frequency characteristics, which can effectively capture high-frequency noise patterns in the image.

[0187] The generation process of the noise image R is as follows:

[0188] Convert the input image I to a grayscale image ;

[0189] right Apply each SRM filter and obtain 30 filtering results;

[0190] Take the absolute value of the filtering results and concatenate them in the channel dimension to form a 30-channel noise feature map;

[0191] The number of channels is adjusted to 3 through 1×1 convolution to obtain a noisy image R with the same number of channels as the original image.

[0192] The SRM filter is designed based on noise residual analysis theory. It can capture the subtle noise fingerprints introduced by image tampering, helping the model identify tampered areas. Experimental results show that the introduction of the noise flow improves the model's F1 score by 6.2% under Gaussian blur (radius 0.5), the IoU by 5.7% under JPEG compression (quality 50), and the precision by 7.3% under Gaussian noise (variance 15), demonstrating the significant contribution of noisy images to improving model performance.

[0193] This invention also provides a tenth embodiment. Building on the foundation of the first embodiment, this embodiment details the model's inference phase. This phase supports dynamic input resizing, using interpolation preprocessing to unify remote sensing images of varying resolutions to 352×352 (Post-FakeV) or 512×512 (Post-FakeL). These resizing is then restored to their original size upon output, ensuring the ability to detect tampering with high-resolution remote sensing images.

[0194] The reasoning process is as follows:

[0195] Preprocessing: Input image Perform normalization (scaling pixel values to [0,1]) and resizing (bilinear interpolation to a standard size);

[0196] Noise image generation: Generate a noisy image through the SRM filter ;

[0197] Model inference: I and R are input into the model, and after dual-stream feature encoding, DTSEM calibration and PFD decoding, the tampering mask prediction M is obtained;

[0198] Size restoration: restore M to the original image size through bilinear interpolation.

[0199] Inference optimization strategy:

[0200] Batch inference: Split large images into overlapping small blocks, and then weight the fusion results after inference to avoid boundary discontinuities.

[0201] Model quantization: Parameters are converted from 32-bit floating point numbers to 16-bit / 8-bit to reduce memory usage and computational complexity;

[0202] Self-attention cache: caches the key-value pairs of the Transformer layer to avoid repeated calculations;

[0203] Multi-scale testing: The input image is generated into 0.75×, 1×, and 1.25× scale versions, and the inference results are averaged to improve robustness.

[0204] Experimental results show that the optimized inference speed is increased by 2.3 times, memory usage is reduced by 35%, and multi-scale testing increases the F1 score by 1.8%, especially for complex scenes and small target detection.

[0205] The present invention provides a method for robust tampering location of remote sensing images based on a segmentation-all-model (SAM). There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications should also be considered within the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A robust tampering localization method for remote sensing images based on the Segmentation Everything Model (SAM), characterized by: The following steps are involved: Step 1, feature input and preprocessing: obtain input remote sensing image , where H and W represent the height and width of the image respectively, and the number of channels of the image is 3; the corresponding noise image is generated by the rich steganalysis model SRM filter , construct a dual-stream input containing optical information and high-frequency noise information, where represents the real number space; Step 2, dual-stream feature encoding: The remote sensing image I and the noise image R are input into the optical stream SAM encoder and the noise stream SAM encoder respectively. The optical stream SAM encoder and the noise stream SAM encoder form a dual-stream SAM encoder with shared parameters. The optical flow SAM encoder and the noise flow SAM encoder have the same structure, both of which contain 4 modules, with the module number represented by i. ; All four modules adopt the same structural design. Each module consists of an adaptive multi-scale feature adapter (AMFA) and a cascade of Transformer layers. The AMFA processes input features using four sets of convolutional kernels of different scales to capture feature information at different scales. The Transformer layer uses a self-attention mechanism to globally model features and enhance feature expression capabilities. The optical flow SAM encoder and the noise flow SAM encoder output multi-level features respectively and ,in It is the multi-level semantic feature output by the optical flow after passing through the i-th module in the optical flow SAM encoder. Optical represents optics and corresponds to the semantic feature extracted from the remote sensing image input. It is the multi-level tampering noise feature output by the noise stream after passing through the i-th module in the noise stream SAM encoder, where noise represents noise, corresponding to the tampering noise feature extracted from the noise image input; Step 3: Dynamic tampering signal enhancement: Through the dynamic tampering signal enhancement module DTSEM, the multi-level tampering noise characteristics are used Multi-level semantic features Perform dynamic calibration and calculate feature weights through a gating mechanism , output calibration characteristics , where Conv represents the convolution operation, Convolution features of noise streams are used to enhance effective tampering signals and suppress texture interference; Step 4, parallel branch forged decoding: the calibration feature Input to a parallel branch forged decoder PFD; the parallel branch forged decoder PFD includes four parallel branches, namely a first branch, a second branch, a third branch and a fourth branch; The first branch includes 1×1 convolution, regular convolution layer, normalization layer, activation function layer and calculation layer for deep supervision; The second branch includes a 3×3 dilated convolution with a dilation rate of 6, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision; The third branch includes a 3×3 dilated convolution with a dilation rate of 12, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision; The fourth branch includes a 3×3 dilated convolution with a dilation rate of 18, a regular convolution layer, a normalization layer, an activation function layer, and a computation layer for deep supervision; The multi-scale tampering features are extracted through convolution branches with different expansion rates, and the pixel-level classification of each layer output in the parallel branch forgery decoder PFD is combined with the deep supervision mechanism, and finally the tampering area mask that integrates multi-scale information is output. ; Based on steps 1 to 4, the construction of the dual-stream SAM model was completed; Step 5, loss function optimization: adopt the combined loss of weighted intersection-over-union loss wIoU and weighted binary cross entropy loss wBCE; Step 6: The two-stream SAM model performs reasoning phase processing: In the reasoning phase, the two-stream SAM model supports dynamic input size.

2. The method according to claim 1, characterized in that In step 1, a 3×3 SRM filter bank is applied to the remote sensing image I to extract the residual signal containing the high-frequency noise pattern introduced by the tampering. The formula is: ， The SRM filter bank contains edge detection kernels in horizontal, vertical and diagonal directions, which is used to explicitly model the noise inconsistencies in the image acquisition and tampering process.

3. The method according to claim 2, characterized in that In step 2, the adaptive multi-scale feature adapter AMFA includes a multi-scale feature extraction unit and a hierarchical interactive fusion unit; The adaptive multi-scale feature adapter AMFA performs the following operations: Input features First, the linear projection layer is used to reduce the dimension, and the low-dimensional features are obtained through GeLU activation and reshaping operations. , the formula is: ， in is the dimension reduction matrix, RS is the reshaping operation; Subsequently, the features are subjected to three 3×3 dilated convolutions with different dilation rates of 6, 12, and 18 to extract multi-scale information, and global context features are obtained through global average pooling, which is expressed as: ， in represents 1×1 convolution, It is a dilated convolution with expansion rates of 6, 12, and 18, corresponding to ; AAP is the adaptive average pooling layer, and Up is the bilinear interpolation upsampling; finally, the multi-scale features are concatenated with the original feature channels to form a feature set containing multi-scale information; The hierarchical interactive fusion unit performs cross-layer fusion of multi-scale features, and the formula is: ， in, Indicates the first level under the first level after hierarchical interaction fusion Features is the jth original multi-scale feature at the lth level; is the activation function, used to introduce nonlinearity; Represents a 3×3 convolution operation; The feature aggregation and recovery unit integrates the multi-scale features after hierarchical interaction and restores the dimensions. The formula is: ， ， in, It represents the aggregated features obtained at level l after feature processing, which is the overall feature representation after comprehensive processing of multiple related features at this level; It is a linear projection layer that restores the feature dimension to the original input dimension C; It is the final output feature obtained after feature processing and dimension restoration at the lth level.

4. The method according to claim 3, characterized in that In step 2, the Transformer layer adopts a hierarchical feature extraction architecture. In the optical stream SAM encoder and the noise stream SAM encoder, the first and second modules are low-level modules, and the third and fourth modules are high-level modules. The low-level modules focus on local texture differences, the high-level modules capture the global semantic distribution, and the representation migration from semantic features to tampering features is achieved through the adaptive multi-scale feature adapter AMFA.

5. The method according to claim 4, characterized in that In step 3, the gating mechanism includes: Multi-level tampering noise features Perform convolution, batch normalization, and PReLU activation operations to generate transition features ; Establish a two-layer feature transformation network with the same structure, and perform convolution, batch normalization, and PReLU activation operations respectively; Transition Features Multi-level semantic features After channel splicing, the two-layer feature transformation network is input, and after two rounds of convolution, batch normalization, PReLU activation operations, and then global average pooling GAP operations, the gate value is generated. ; The global average pooling operation performs an average calculation on the feature map in the spatial dimension to obtain a feature vector of fixed length.

6. The method according to claim 5, characterized in that In step 4, the first branch, second branch, third branch and fourth branch are used to capture local detail features, medium-scale features, larger-scale features and global context features, respectively, and through jump connections, the shallow feature map that retains high-resolution details and the deep feature map containing abstract semantic information are fused by channel concatenation or weighted summation.

7. The method according to claim 6, characterized in that In step 5, the weighting factor in the loss function is designed based on the local neighborhood mean. : ， in is located at image coordinates The weighting factor of the pixel in the loss function is used to adjust the weight of the pixel in the loss calculation; True mask Center coordinates Pixel Neighborhood mean; is the coordinate in the real mask G The pixel value at .

8. The method according to claim 7, characterized in that In step 6, the dual-stream SAM model will first adjust the size of the remote sensing images uniformly by interpolation preprocessing for the input remote sensing images of different resolutions. For the images of the Post-FakeV dataset, the size is uniformly adjusted to Size; for images in the Post-FakeL dataset, unified to Size; The two-stream SAM model performs inference calculation based on the adjusted image, and after obtaining the result, restores the output to the original size of the image through interpolation operation.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 8 are executed.

Images