Image tamper detection method and system based on multi-view feature adaptive fusion

Abstract

The application discloses a kind of based on multi-view feature adaptive fusion image tampering detection method and system, belong to the cross technical field of computer vision and information security, including: obtaining the text image to be detected;Extract shallow visual features using the shallow layer of backbone network;Respectively extract the frequency domain feature, noise domain feature and principal component domain feature of text image using auxiliary view encoder;Using multi-view arccosine similarity weighted fusion module, the shallow visual features are cascaded with the above three features Adaptive fusion, each fusion is according to the similarity between current fusion result and the feature to be fused Weight distribution, weight and similarity are negatively correlated;According to the fusion feature, generate tampering prediction mask.The present application first introduces principal component analysis to highlight weak tampering traces, suppress feature redundancy in multi-view fusion, amplify key tampering clues through arccosine similarity weighted adaptive, improve precision and robustness, light in calculation, suitable for real scene text forensics.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and information security, and in particular relates to an image tampering detection method and system based on multi-view feature adaptive fusion. Background Technology

[0002] With the rapid development of image editing technologies (such as Adobe Photoshop) and generative artificial intelligence, text and image tampering has become increasingly easy and difficult to detect, posing a serious threat to information security in government, judicial, and financial sectors. Common text tampering operations include erasing, splicing, copying-moving, and reprinting.

[0003] Existing image tampering detection methods primarily rely on single RGB visual features or simple frequency domain features. However, text images have unique characteristics: tampered text and background often exhibit extremely high visual consistency, making it difficult for single RGB visual features to capture subtle tampering artifacts. To address this issue, some studies have attempted to introduce multi-view approaches (such as noise streams and frequency domain streams) to obtain supplementary clues. However, these methods generally suffer from a key problem: when fusing multi-view features, they typically employ simple feature concatenation or direct addition strategies. This coarse fusion approach cannot effectively handle the high similarity and information redundancy between features from different views, making the model susceptible to interference from redundant features during training. This masks the subtle features that truly contain tampering clues, thus limiting the accuracy, robustness, and generalization ability of tampering detection to complex real-world scenarios. Summary of the Invention

[0004] To address the problems of single feature extraction dimension and severe information redundancy in multi-view fusion in existing technologies, this invention proposes an image tampering detection method and system based on adaptive fusion of multi-view features to solve the aforementioned problems in existing technologies.

[0005] To achieve the above objectives, this invention provides an image tampering detection method based on multi-view feature adaptive fusion, comprising:

[0006] Obtain the image of the text to be detected;

[0007] The shallow visual features of the text image are extracted using the shallow layers of the backbone network;

[0008] The frequency domain features, noise domain features, and principal component domain features of the text image are extracted using an auxiliary view encoder.

[0009] The shallow visual features are cascaded adaptively fused with the frequency domain features, the noise domain features, and the principal component domain features using a multi-view inverse cosine similarity weighted fusion module to obtain fused features; wherein, the cascaded adaptive fusion includes multiple fusions, and each fusion dynamically assigns weights based on the inverse cosine similarity between the current fusion result and the feature to be fused;

[0010] A tamper prediction mask is generated based on the fusion features.

[0011] Optionally, the auxiliary view encoder includes a parallel frequency domain sensing head, a noise sensing head, and a principal component sensing head; the frequency domain sensing head is used to extract the frequency domain features, the noise sensing head is used to extract the noise domain features, and the principal component sensing head is used to extract the principal component domain features.

[0012] Optionally, the process of extracting frequency domain features by the frequency domain sensing head includes:

[0013] The text image is converted to the YCbCr color space, and the discrete cosine transform coefficients of the Y channel are extracted. A JPEG quantization table is obtained, and the discrete cosine transform coefficients are one-hot encoded and then processed by a convolution module. The result is multiplied by the JPEG quantization table expanded to the size of the text image to obtain a frequency representation. The frequency representation is then input into a feature extraction network to obtain the frequency domain features.

[0014] Optionally, the noise-sensing head filters the text image through a Bayar convolutional layer to obtain a noise view that suppresses color and texture content, and inputs the noise view into a feature extraction network to obtain the noise domain features.

[0015] Optionally, the process of extracting principal component domain features by the principal component sensing head includes:

[0016] The text image is histogram equalized and flattened into a two-dimensional matrix. The first principal component is extracted by singular value decomposition. The image is then reconstructed and converted into a grayscale image through inverse transformation. The global intensity distribution representation is copied in the channel dimension and then input into a feature extraction network to obtain the principal component domain features.

[0017] Optionally, the multi-view inverse cosine similarity weighted fusion module includes multiple cascaded basic fusion units; each basic fusion unit is designed for any two features to be fused. and Calculate the cosine similarity between the two, and obtain the inverse cosine similarity weighting factor. Based on the inverse cosine similarity weighting factor, for any two features to be fused and Perform feature fusion.

[0018] Optionally, the shallow visual features are used as... The frequency domain features are used as Input the first basic fusion unit to obtain the first fusion result; use the first fusion result as... The principal component domain features are used as Input the second basic fusion unit to obtain the second fusion result; use the second fusion result as... The noise domain features are used as Input the third basic fusion unit and output the final fusion feature.

[0019] Optionally, the process of generating a tamper prediction mask based on the fused features includes:

[0020] The fused features are input into the deep layers of the backbone network for feature enhancement to obtain high-level tampering features; the shallow visual features, the fused features, and the high-level tampering features are upsampled to the same size and then concatenated to obtain multi-scale features; the multi-scale features are input into a convolutional decoder for decoding to output a pixel-level tampering prediction mask.

[0021] Optionally, the method further includes: based on the text image tampering detection dataset, using a weighted combination of cross-entropy loss and Dice loss as the total loss function, to perform end-to-end joint training on the backbone network, the auxiliary view encoder, the multi-view inverse cosine similarity weighted fusion module, and the convolutional decoder.

[0022] The present invention also provides an image tampering detection system based on multi-view feature adaptive fusion, for implementing the above method, comprising:

[0023] The backbone network, whose shallow layers are used to extract shallow visual features of the input text image;

[0024] An auxiliary view encoder is used to extract the frequency domain features, noise domain features, and principal component domain features of the text image, respectively.

[0025] The multi-view inverse cosine similarity weighted fusion module is used to perform cascaded adaptive fusion of the shallow visual features with the frequency domain features, the noise domain features and the principal component domain features to obtain fused features; wherein, the cascaded adaptive fusion includes multiple fusions, and each fusion dynamically assigns weights based on the inverse cosine similarity between the current fusion result and the feature to be fused, and the weights are negatively correlated with the inverse cosine similarity;

[0026] A convolutional decoder is used to generate a tamper prediction mask based on the fused features.

[0027] Compared with the prior art, the present invention has the following advantages and technical effects:

[0028] This invention provides an image tampering detection method based on multi-view feature adaptive fusion. First, an auxiliary view encoder is used to extract frequency domain, noise domain, and principal component domain features in parallel. The principal component view, introduced for the first time, highlights the disruption of the global intensity distribution continuity caused by tampering, making subtle traces invisible in the RGB domain visible. Second, a multi-view inverse cosine similarity weighted fusion module is used for cascaded fusion. The weights are negatively correlated with similarity, adaptively suppressing redundant information and preventing key clues from being obscured. Moreover, it only involves cosine similarity calculation and scalar multiplication, making the computation lightweight. Finally, end-to-end joint training enables the model to automatically learn the optimal weight allocation, exhibiting strong robustness to compression, blurring, and other degradations in real-world scenes, significantly improving the accuracy and generalization ability of tampering detection. Attached Figure Description

[0029] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 This is a flowchart illustrating the technical solution of an embodiment of the present invention;

[0031] Figure 2 This is a flowchart of the basic fusion unit in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the multi-view inverse cosine similarity weighted fusion module according to an embodiment of the present invention;

[0033] Figure 4 This is a diagram illustrating the overall architecture of the model design in an embodiment of the present invention. Detailed Implementation

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0036] Example 1

[0037] This embodiment provides an image tampering detection method based on multi-view feature adaptive fusion. This method extracts multi-view complementary tampering clues in the frequency domain, noise domain, and principal component domain through an auxiliary view encoder, and adaptively suppresses redundant features using a multi-view inverse cosine similarity weighted fusion module to achieve accurate localization of weakly tampered areas.

[0038] The detailed steps of the method are as follows: (1) Data preparation and preprocessing: Obtain the tampered text image data and the corresponding real tampering mask label, and perform data augmentation; (2) Basic visual feature extraction: Use the main feature extraction backbone network to extract the shallow visual features of the image; (3) Auxiliary view feature extraction: Use the auxiliary view encoder (AVE) to extract multi-dimensional complementary tampering features from the frequency domain, noise domain and principal component domain; (4) Adaptive feature fusion: Use the multi-view inverse cosine similarity weighted fusion module (MICSF) to perform redundancy fusion of visual features and auxiliary features; (5) Model decoding and training: Input the fused features into the decoder to obtain the prediction mask, calculate the prediction loss, and finally use it for tampering inference of actual images. The flowchart of the technical solution is as follows. Figure 1 As shown.

[0039] Furthermore, step (1) includes the following specific contents:

[0040] (1-1) Image-mask label pairs are selected from multiple available text image tampering detection datasets, and the mask... In this example, a pixel value of 1 represents a tampered area, and 0 represents a genuine area. This embodiment focuses on detecting tampered text in financial scenarios, such as modifications or additions to numbers and sensitive content on invoices, forms, and promotional posters. Detecting text tampering in such images is highly concealed and difficult.

[0041] (1-2) Preprocess the training images and their corresponding masks, including cropping or scaling the images to a fixed size (e.g., 512×512) and normalizing them.

[0042] (1-3) Perform data augmentation operations, including random horizontal / vertical flipping, random rotation, random JPEG compression, etc., to simulate the degradation process of images in real-world transmission on social networks.

[0043] Furthermore, step (2) includes the following specific contents:

[0044] Preprocessed RGB text image The input is fed into the shallow layers of the backbone network (such as the first two stages of the ConvNeXt network) to extract shallow visual features of the image. and .

[0045] Furthermore, step (3) includes the following specific contents:

[0046] The Auxiliary View Encoder (AVE) proposed in this invention consists of three parallel sensing heads, as detailed below:

[0047] (3-1) Frequency domain sensor head: Converts RGB images to YCbCr color space and extracts the discrete cosine transform (DCT) coefficients of the Y channel. and obtain the JPEG quantization table. After one-hot encoding of the DCT coefficients, they are processed through two convolutional modules (consisting of 3×3 convolutional layers, batch normalization layers, and ReLU activation layers) and extended. The frequency characterization is obtained by multiplying the JPEG quantization table of the size. Finally, Y is input into the feature extraction network to obtain frequency domain features. .

[0048] (3-2) Noise-sensing head: The image is filtered using Bayar convolutional layers to suppress color and texture content and extract noise views that highlight tampering artifacts. Then it is input into a feature extraction network to obtain noise features. .

[0049] (3-3) Principal Component Perceptron: Performs histogram equalization on the image and flattens it into a shape of... The image is a two-dimensional matrix. Singular value decomposition (SVD) is applied to extract the first principal component of the image, which is then reconstructed and converted to a grayscale image via inverse transform. The global intensity distribution representation is then obtained by copying this representation along the channel dimension. Subsequently, it is input into a feature extraction network to obtain principal component features. .

[0050] The feature extraction network mentioned refers to the first two stages of the ConvNeXt network. By using ConvNeXt, an excellent convolutional neural network, for feature extraction, the model can pay attention to the potential inconsistencies in local features introduced by the tampering operation in different views.

[0051] Of the three auxiliary views used in this embodiment, the frequency view and noise view have been widely and effectively applied in natural image tampering detection and tampered text detection. This invention, however, is the first to utilize the principal component view for tampered text detection and has proven its effectiveness. In tampered receipts or forms with clean backgrounds, the RGB view typically struggles to detect valid forgery clues. Tampering operations generally disrupt the global intensity distribution of the image, making tampering traces more prominent in the principal component view, such as the edges of the tampering operation.

[0052] Furthermore, step (4) includes the following specific contents:

[0053] The Multi-View Inverse Cosine Similarity Weighted Fusion Module (MICSF) proposed in this invention consists of three cascaded Basic Fusion Units (BFUs) used to fuse visual features With auxiliary view features ( , , Adaptive fusion is performed. The flowchart of the basic fusion unit is as follows: Figure 2 As shown.

[0054] (4-1) For any two features to be fused and The Basic Fusion Unit (BFU) first calculates the cosine similarity between the two components and then obtains the weighting factor for the inverse cosine similarity. ,in It is the cosine similarity calculation function for features.

[0055] (4-2) Feature fusion based on weighting factors: ,in The scaling factor is set to 10 in this embodiment based on experience. This mechanism assigns higher weights to features with low similarity and lower weights to features with high similarity, thereby effectively reducing information redundancy in the feature fusion process.

[0056] (4-3) For example Figure 3 As shown, this embodiment uses three cascaded basic fusion units to sequentially... , and Integration In the end, the fused features are output. In this process, the fusion of each new auxiliary view feature is strictly based on the previous fusion result and iterated until all views have been processed.

[0057] The reason this invention employs a cascaded rather than parallel fusion approach (the parallel approach calculates the similarity of each auxiliary view feature with the basic visual features separately, fuses them, and then sums all the branch results) is that the parallel approach only considers the redundancy between a single auxiliary view and the basic view, while ignoring the potentially severe information overlap between different auxiliary views (such as the frequency domain and the noise domain). In contrast, the cascaded fusion approach of this invention can accumulate the extracted effective information. When a new auxiliary view is introduced, the inverse cosine similarity calculation is based on the "current view" and the "fused view containing all historical effective information." This design can globally reduce the information redundancy between the current auxiliary view and the overall fused view, forcing the network to only "absorb" the unique and complementary tampering clues in the current view at each iteration, thereby constructing a more discriminative and robust comprehensive tampering feature representation.

[0058] Furthermore, step (5) includes the following specific contents:

[0059] (5-1) Model Decoding: fusing features The data is fed into deeper layers of the backbone network (such as the last two stages of the ConvNeXt network) for further feature enhancement to obtain high-level tampering features. and Finally, the result obtained in step (2) The result obtained in step (4) as well as and All adjusted to Multi-scale features are obtained by stitching together features after scaling them. The feature is then decoded by a convolutional decoder, outputting a pixel-level tamper prediction mask. The decoding process can be formalized as follows:

[0060] ;

[0061] ;

[0062] in, It is a convolutional encoder, which consists of 3×3 convolutional layers, batch normalization layers, ReLU activation layers, and 1×1 convolutional layers. For feature splicing operations, This represents bilinear interpolation that magnifies the spatial resolution of the feature map by a factor of x.

[0063] Therefore, the overall model proposed in this invention is as follows: Figure 4 As shown, it includes a backbone network, an auxiliary view encoder, a multi-view inverse cosine similarity weighted fusion module, and a convolutional decoder:

[0064] Backbone network: shallow layers are used to propose basic visual features, and deep layers are used to further enhance the fused features.

[0065] Auxiliary view encoder: used to extract multi-dimensional complementary tampering features from the frequency domain, noise domain, and principal component domain.

[0066] Multi-view inverse cosine similarity weighted fusion module: used to calculate the inverse cosine similarity between features of different views, and adaptively assign weights based on similarity to complete the cascade fusion of features.

[0067] Convolutional decoder: Used to decode the fused features and output the final tamper location mask.

[0068] (5-2) Model Training: Based on the obtained text image tampering detection dataset, we perform end-to-end joint training on the entire network. The specific training parameters are as follows:

[0069] a) Number of iterations: 200;

[0070] b) Image resolution: 512×512;

[0071] c) Optimizer: AdamW;

[0072] d) Learning rate: Initially 0.0001, eventually decreasing to 0.000001 according to the cosine annealing strategy;

[0073] e) Weight decay: 0.0005;

[0074] f) Batch size: 12;

[0075] g) Loss function: Cross-entropy loss With Dice loss The weighted combination, the total loss during model training + The weight .

[0076] (5-3) Inference and Evaluation: Inputting the text image to be tested into the trained model yields a prediction mask. For high-resolution images, a 512×512 sliding window is used for inference and the results are stitched together.

[0077] As demonstrated by the above embodiments, this embodiment targets the scenario of detecting tampered text in financial document images, and is particularly suitable for financial document images with relatively clean backgrounds, high resolution, and regular layout structures, such as bills, transaction lists, payment vouchers, financial contracts, bank statements, insurance policies, and promotional materials for wealth management products. In such scenarios, small-scale sensitive text such as amounts, account numbers, dates, quantities, names, and transaction numbers, after being tampered with, usually only produce extremely subtle local artifacts, which are difficult for the human eye to detect directly. To address this, this embodiment introduces a principal component analysis view to highlight the discontinuity in intensity distribution caused by tampering operations in the financial document image. This view, along with other auxiliary views, serves as supplementary clues to the RGB view. Adaptive feature fusion is performed through a multi-view inverse cosine similarity weighted fusion module, thereby effectively reducing conflicts and redundancy between features from different views, enhancing the model's ability to perceive small-scale tampered text regions, and thus improving the detection effect of tampered text in financial document images.

[0078] Example 2

[0079] This embodiment provides an image tampering detection method based on multi-view feature adaptive fusion, including:

[0080] Obtain the image of the text to be detected;

[0081] The shallow visual features of text images are extracted using the shallow layers of the backbone network;

[0082] The frequency domain features, noise domain features, and principal component domain features of the text image are extracted using an auxiliary view encoder.

[0083] The auxiliary view encoder includes a parallel frequency domain sensing head, a noise sensing head, and a principal component sensing head; the frequency domain sensing head is used to extract frequency domain features, the noise sensing head is used to extract noise domain features, and the principal component sensing head is used to extract principal component domain features.

[0084] The process of extracting frequency domain features using a frequency domain sensing head includes:

[0085] The text image is converted to the YCbCr color space, and the discrete cosine transform coefficients of the Y channel are extracted. The JPEG quantization table is obtained, and the discrete cosine transform coefficients are one-hot encoded and then processed by a convolution module. The result is multiplied by the JPEG quantization table expanded to the size of the text image to obtain the frequency representation. The frequency representation is then input into the feature extraction network to obtain the frequency domain features.

[0086] The noise-sensing head filters the text image through Bayar convolutional layers to obtain a noise view that suppresses color and texture content. The noise view is then input into a feature extraction network to obtain noise domain features.

[0087] The process of extracting principal component domain features using a principal component perceptron includes:

[0088] The text image is histogram equalized and flattened into a two-dimensional matrix. The first principal component is extracted by singular value decomposition. The image is then reconstructed and converted into a grayscale image through inverse transformation. The global intensity distribution representation is copied in the channel dimension and then input into the feature extraction network to obtain the principal component domain features.

[0089] A multi-view inverse cosine similarity weighted fusion module is used to perform cascaded adaptive fusion of shallow visual features with frequency domain features, noise domain features and principal component domain features to obtain fused features. The cascaded adaptive fusion includes multiple fusions, and each fusion dynamically assigns weights based on the inverse cosine similarity between the current fusion result and the feature to be fused.

[0090] The multi-view inverse cosine similarity weighted fusion module includes multiple cascaded basic fusion units; each basic fusion unit is designed for any two features to be fused. and Calculate the cosine similarity between the two, and obtain the inverse cosine similarity weighting factor. and through Output a new fusion result, in which, This is the scaling factor.

[0091] Using shallow visual features as Frequency domain characteristics as Input the first basic fusion unit to obtain the first fusion result; use the first fusion result as... Principal component domain features as Input the second basic fusion unit to obtain the second fusion result; use the second fusion result as... Noise domain characteristics as Input the third basic fusion unit and output the final fusion feature.

[0092] A tamper prediction mask is generated based on the fused features. The process includes: inputting the fused features into the deep layers of the backbone network for feature enhancement to obtain high-level tamper features; upsampling the shallow visual features, fused features, and high-level tamper features to the same size and then concatenating the features to obtain multi-scale features; inputting the multi-scale features into a convolutional decoder for decoding and outputting a pixel-level tamper prediction mask.

[0093] The method also includes: based on the text image tampering detection dataset, using a weighted combination of cross-entropy loss and Dice loss as the total loss function, and performing end-to-end joint training on the backbone network, auxiliary view encoder, multi-view inverse cosine similarity weighted fusion module and convolutional decoder.

[0094] This embodiment also provides an image tampering detection system based on multi-view feature adaptive fusion, including:

[0095] The backbone network, whose shallow layers are used to extract shallow visual features of the input text image;

[0096] An auxiliary view encoder is used to extract frequency domain features, noise domain features, and principal component domain features of the text image, respectively.

[0097] The multi-view inverse cosine similarity weighted fusion module is used to perform cascaded adaptive fusion of shallow visual features with frequency domain features, noise domain features and principal component domain features to obtain fused features. The cascaded adaptive fusion includes multiple fusions. Each fusion dynamically assigns weights based on the inverse cosine similarity between the current fusion result and the feature to be fused. The higher the similarity, the lower the weight is assigned.

[0098] A convolutional decoder is used to generate a tamper prediction mask based on fused features.

[0099] To address the problem that highly consistent tampered text with the background in text images makes effective detection by a single visual feature difficult, this invention introduces an auxiliary view encoder. This encoder includes a frequency domain perceptron, a noise perceptron, and a principal component perceptron, capable of extracting complementary tampering clues from three dimensions: Discrete Cosine Transform (DCT), Restricted Bayar Convolution, and Singular Value Decomposition (SVD). Secondly, to address the severe information redundancy problem in existing methods when fusing multi-view features, this invention proposes a multi-view inverse cosine similarity weighted fusion module. This module consists of multiple cascaded basic fusion units, dynamically allocating fusion weights by calculating the inverse cosine similarity between different view features. Specifically, features with higher similarity are assigned lower weights, thereby adaptively suppressing the interference of redundant features and amplifying view features containing key tampering clues. This invention is the first to introduce principal component analysis into tampered text detection to capture subtle tampering traces that are difficult to detect visually, and cleverly solves the feature redundancy problem in multi-view fusion, enabling the model to more accurately capture weak tampering artifacts. This method exhibits stronger robustness and higher detection accuracy when facing complex real-world scenarios and diverse tampering techniques, making it highly practical.

[0100] This invention introduces principal component analysis (PCA) view features for the first time in text tampering detection. These features effectively capture tampering operations, such as copy-move and erase, which disrupt the continuity of the global intensity distribution of the image. This allows subtle tampering traces, imperceptible to the naked eye in the RGB domain, to be revealed, significantly enriching the methods for exposing these subtle tampering artifacts. Experimental results show that this view is crucial for detecting tampered text in documents, significantly enhancing the model's ability to perceive subtle tampering regions and improving the overall effectiveness of document image tampering detection.

[0101] This invention proposes a multi-view inverse cosine similarity weighted fusion module, which solves the feature redundancy problem in multi-view fusion from a new perspective. This module abandons the traditional simple splicing and direct addition methods, using inverse cosine similarity as a dynamic weight to adaptively suppress highly similar redundant features, amplify the influence of views containing key complementary tampering clues, and significantly improve the model's detection accuracy.

[0102] This invention unifies multi-view feature extraction and redundancy removal fusion within an end-to-end deep learning framework, eliminating the need for cumbersome manual rule intervention. Because the model of this invention can efficiently fuse comprehensive cues from the frequency domain, noise domain, and principal component domain, it exhibits stronger robustness and cross-scene generalization when facing image compression, blurring, and other degradation conditions present in real-world scenarios, thus possessing significant practical value.

[0103] This invention proposes a computationally efficient and lightweight multi-view fusion mechanism. Existing multi-view fusion methods often rely on complex attention mechanisms or convolutional layer calculations, which introduce significant computational overhead. The feature fusion method proposed in this invention achieves adaptive weighting through simple cosine similarity calculation and scalar multiplication. This design ensures efficient feature redundancy removal fusion with almost no additional computation or model parameters, making it highly suitable for deployment in practical high-resolution text image detection tasks.

[0104] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image tampering detection method based on multi-view feature adaptive fusion, characterized in that, Includes the following steps: Obtain the image of the text to be detected; The shallow visual features of the text image are extracted using the shallow layers of the backbone network; The frequency domain features, noise domain features, and principal component domain features of the text image are extracted using an auxiliary view encoder. The auxiliary view encoder includes a parallel frequency domain sensing head, a noise sensing head, and a principal component sensing head; the frequency domain sensing head is used to extract the frequency domain features, the noise sensing head is used to extract the noise domain features, and the principal component sensing head is used to extract the principal component domain features. The shallow visual features are cascaded adaptively fused with the frequency domain features, the noise domain features, and the principal component domain features using a multi-view inverse cosine similarity weighted fusion module to obtain fused features; wherein, the cascaded adaptive fusion includes multiple fusions, and each fusion dynamically assigns weights based on the inverse cosine similarity between the current fusion result and the feature to be fused; The multi-view inverse cosine similarity weighted fusion module includes multiple cascaded basic fusion units; each basic fusion unit is designed for any two features to be fused. and Calculate the cosine similarity between the two, and obtain the inverse cosine similarity weighting factor. Based on the inverse cosine similarity weighting factor, for any two features to be fused and Perform feature fusion; Using the shallow visual features as The frequency domain features are used as Input the first basic fusion unit to obtain the first fusion result; use the first fusion result as... The principal component domain features are used as Input the second basic fusion unit to obtain the second fusion result; use the second fusion result as... The noise domain features are used as Input the third basic fusion unit and output the final fusion feature; A tamper prediction mask is generated based on the fusion features.

2. The image tampering detection method based on multi-view feature adaptive fusion according to claim 1, characterized in that, The process of extracting frequency domain features by the frequency domain sensing head includes: The text image is converted to the YCbCr color space, and the discrete cosine transform coefficients of the Y channel are extracted. A JPEG quantization table is obtained, and the discrete cosine transform coefficients are one-hot encoded and then processed by a convolution module. The result is multiplied by the JPEG quantization table expanded to the size of the text image to obtain a frequency representation. The frequency representation is then input into a feature extraction network to obtain the frequency domain features.

3. The image tampering detection method based on multi-view feature adaptive fusion according to claim 1, characterized in that, The noise-sensing head filters the text image through a Bayar convolutional layer to obtain a noise view that suppresses color and texture content. The noise view is then input into a feature extraction network to obtain the noise domain features.

4. The image tampering detection method based on multi-view feature adaptive fusion according to claim 1, characterized in that, The process of extracting principal component domain features using the principal component perceptron includes: The text image is histogram equalized and flattened into a two-dimensional matrix. The first principal component is extracted by singular value decomposition. The image is then reconstructed and converted into a grayscale image through inverse transformation. The global intensity distribution representation is copied in the channel dimension and then input into a feature extraction network to obtain the principal component domain features.

5. The image tampering detection method based on multi-view feature adaptive fusion according to claim 1, characterized in that, The process of generating a tamper prediction mask based on fusion features includes: The fused features are input into the deep layers of the backbone network for feature enhancement to obtain high-level tampering features; the shallow visual features, the fused features, and the high-level tampering features are upsampled to the same size and then concatenated to obtain multi-scale features; the multi-scale features are input into a convolutional decoder for decoding to output a pixel-level tampering prediction mask.

6. The image tampering detection method based on multi-view feature adaptive fusion according to claim 1, characterized in that, The method also includes: based on the text image tampering detection dataset, using a weighted combination of cross-entropy loss and Dice loss as the total loss function, performing end-to-end joint training on the backbone network, the auxiliary view encoder, the multi-view inverse cosine similarity weighted fusion module, and the convolutional decoder.

7. An image tampering detection system based on multi-view feature adaptive fusion, used to implement the method according to any one of claims 1-6, characterized in that, include: The backbone network, whose shallow layers are used to extract shallow visual features of the input text image; An auxiliary view encoder is used to extract the frequency domain features, noise domain features, and principal component domain features of the text image, respectively. The multi-view inverse cosine similarity weighted fusion module is used to perform cascaded adaptive fusion of the shallow visual features with the frequency domain features, the noise domain features and the principal component domain features to obtain fused features; wherein, the cascaded adaptive fusion includes multiple fusions, and each fusion dynamically assigns weights based on the inverse cosine similarity between the current fusion result and the feature to be fused, and the weights are negatively correlated with the inverse cosine similarity; A convolutional decoder is used to generate a tamper prediction mask based on the fused features.

Images