Image re-coloring detection method and system based on multi-domain fusion

Abstract

The invention relates to the technical field of image tampering detection, in particular to an image re-coloring detection method and system based on multi-domain fusion. According to the method, color domain, residual domain and illumination image shallow layer features are extracted from images, then splicing is performed, multi-domain features are input into ViT models of different sizes, multi-scale deep layer features are extracted by using a self-attention mechanism, and the extracted multi-scale deep layer features are subjected to up-sampling and gradual fusion, so that the multi-scale deep layer features are obtained. Generating a global feature map to realize positioning and detection of a re-colored image; according to the multi-domain fusion detection method, the accuracy of image re-coloring detection is effectively improved, and compared with a traditional single-domain detection method, the multi-domain fusion detection method has the advantages that tiny changes of the image between different domains can be better captured, so that the detection robustness is improved, re-coloring areas can be marked, and the multi-domain fusion detection method has better application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of image tampering detection, and in particular to an image recoloring detection method and system based on multi-domain fusion. Background Art

[0002] Image recoloring is an emerging image editing technology that changes the color style of an image by tampering with pixel values. It is widely used in design, advertising, entertainment and other fields. With the popularity of various image editing software and artificial intelligence, anyone can recolor images at a very low cost. Such tampered images are visually difficult to be recognized by the human eye. However, this technology also brings the potential risk of image tampering. Especially in the fields of news, law, public security, etc., the need for image authenticity detection is becoming increasingly urgent. Therefore, it is particularly important to develop an efficient and accurate image recoloring detection technology.

[0003] The most popular detection method currently is to look for some specific artifacts related to camera characteristics or caused by specific editing behaviors, which mainly rely on traditional image processing methods such as color histogram comparison, feature matching or image texture analysis to identify the authenticity of the image. However, these methods often do not work well when processing complex images and high-quality recolored images. In recent years, image processing methods based on deep learning have shown good performance, but existing technologies still face the following challenges:

[0004] Insufficient fusion of multi-domain features: Existing technologies usually perform analysis in a single domain (such as color domain or frequency domain). However, image recoloring operations are more covert than other tampering techniques, and information from a single domain is often insufficient to accurately identify recoloring behavior.

[0005] Weak robustness and generalization of the model: Existing methods are unstable in different scenes and different types of images. They may be able to cope with images generated by known image recoloring methods, but new image recoloring technologies are emerging like mushrooms after rain. Therefore, it is difficult to handle image recoloring in complex and unknown real scenes.

[0006] Unable to locate recolored areas: Existing methods can only identify whether an image has been tampered with by recoloring, but cannot mark possible recolored areas and cannot help users perform further analysis and confirmation. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the present invention aims to provide an image recolorization detection method and system based on multi-domain fusion, so as to solve the problem that image recolorization detection is inaccurate with information from a single domain or unstable in detection of images in different scenes and of different types, and further realize the positioning of the recolorization area.

[0008] In order to solve the above problems, the present invention adopts the following technical solutions:

[0009] The present invention provides an image recoloring detection method based on multi-domain fusion, comprising:

[0010] Get the image and normalize the image size.

[0011] The size-normalized image is converted from the RGB space to the HSV color space to obtain an HSV image for color domain feature extraction.

[0012] The residual domain features of the image after size normalization are extracted through the residual feature extraction module.

[0013] Generate a light map from the size-normalized image.

[0014] A multi-domain fusion feature tensor is constructed based on the HSV image, residual domain features and illumination map of the acquired image.

[0015] The Vision Transformer model is used to extract high, medium and low levels of image feature sequences. The three levels of image feature sequences are concatenated with their corresponding position vectors and input into the encoder composed of the self-attention mechanism and the forward propagation network to extract the multi-scale deep features of the image.

[0016] Based on the multi-scale deep features of the extracted image, a global feature map is generated by gradually upsampling and fusion.

[0017] The global feature map is pooled to fuse features at different positions into a global feature vector. Based on the obtained global feature vector, a classifier is used to perform a binary classification judgment on whether it has been recolored.

[0018] As an implementable method, it also includes outputting a positioning map through a decoder based on the generated global feature map; and determining whether the pixel is tampered with by recoloring based on the size of the pixel value of each pixel in the positioning map.

[0019] As an implementable method, extracting residual domain features of the image by a residual feature extraction module from the size-normalized image includes:

[0020] The normalized image is passed through the residual feature enhancement module to extract the high-frequency information related to recoloring in the image using a high-pass filter kernel and enhanced, and then input into the ResNet18 residual feature extraction module. The high-level residual features are extracted through the ResNet18 residual feature extraction module to obtain the residual domain features of the image.

[0021] As an implementable method, generating a light map from the size-normalized image includes:

[0022] Compute the global mean for each color channel of the image.

[0023] The light source color estimate is obtained by normalizing the global average of each color channel.

[0024] The light map is generated by normalizing the light color estimate at each pixel and applying the light color estimate to each pixel.

[0025] As an implementable method, the use of the Vision Transformer model to extract high, medium and low level image feature sequences includes:

[0026] A 4×4 Patch ViT is used to extract local fine high-level feature sequences. The 4×4 Patch ViT consists of a 6-layer basic Transformer encoder. The size of the input sequence is .

[0027] The 8×8 Patch ViT is used to extract the mid-level feature sequence. The 8×8 Patch ViT consists of 8 layers of basic Transformer encoders. The size of the input sequence is .

[0028] A 16×16 Patch ViT is used to extract the global low-level feature sequence. The 16×16 Patch ViT consists of a 12-layer basic Transformer encoder. The size of the input sequence is .

[0029] As an implementable embodiment, the self-attention mechanism includes:

[0030] The multi-domain fusion feature tensor is transformed through three different linear transformations to generate the Query matrix, Key matrix and Value matrix.

[0031] The attention weight matrix is ​​obtained by calculating the dot product of the Query matrix and the Key matrix and normalizing them.

[0032] According to the weighted summation of attention weights and Value matrix, multi-scale deep features are obtained.

[0033] As an implementable method, the method of generating a global feature map by stepwise upsampling and fusion based on the multi-scale deep features of the extracted image includes:

[0034] from The features output by the patch encoder are upsampled using deconvolution to The features output by the Patch encoder have the same spatial resolution, and the upsampled features are The features output by the patch encoder are fused by element-by-element addition.

[0035] The features after feature fusion are upsampled by using deconvolution to The features output by the Patch encoder have the same spatial resolution, and the upsampled features are fused with the features output by the 4x4 Patch encoder by element-by-element addition.

[0036] The features after feature fusion are upsampled using deconvolution to make their width and height the same as the acquired image, and finally a fused global feature map is generated.

[0037] The present invention provides an image recoloring detection system based on multi-domain fusion, comprising an image acquisition and processing module, a color domain conversion module, a residual domain feature extraction module, an illumination map generation module, a multi-domain fusion feature tensor generation module, a multi-scale deep feature extraction module, a global feature map generation module and a determination module.

[0038] The image acquisition and processing module is used to acquire images and perform size normalization processing on the images.

[0039] The color domain conversion module is used to convert the image after size normalization from the RGB space to the HSV color space to obtain an HSV image for color domain feature extraction.

[0040] The residual domain feature extraction module is used to extract residual domain features of the image after the size normalization through the residual feature extraction module.

[0041] The illumination map generation module is used to generate an illumination map from the image after size normalization.

[0042] The multi-domain fusion feature tensor generation module is used to construct a multi-domain fusion feature tensor according to the HSV image, residual domain features and illumination map of the acquired image.

[0043] The multi-scale deep feature extraction module is used to extract high, medium and low level image feature sequences using the Vision Transformer model, and concatenate the three levels of image feature sequences with their corresponding position vectors respectively, and input them into the encoder composed of the self-attention mechanism and the forward propagation network to extract the multi-scale deep features of the image.

[0044] The global feature map generation module is used to generate a global feature map by gradually upsampling and fusion based on the multi-scale deep features of the extracted image.

[0045] The determination module is used to fuse features at different positions of the global feature map into a global feature vector through global pooling, and based on the obtained global feature vector, a classifier is used to perform a binary classification determination on whether the global feature map has been recolored.

[0046] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the image recoloring detection method based on multi-domain fusion are implemented.

[0047] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image recoloring detection method based on multi-domain fusion are implemented.

[0048] The beneficial effects of the present invention are as follows: through the multi-domain fusion detection method of the present invention, the accuracy of image recolorization detection is effectively improved; compared with the traditional single-domain detection method, the present invention can better capture the slight changes of the image between different domains, thereby improving the robustness of the detection; the multi-scale ViT model further enhances the fusion effect of multi-domain information, can adapt to different image recoloring methods, and has a wide range of applicability; in addition, unlike previous detection methods that can only detect whether the image has been recolored, the present invention can also mark the recolored area, and has a better application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of an image recoloring detection method based on multi-domain fusion in an embodiment of the present invention.

[0050] Figure 2 Schematic diagram of an image recoloring detection system based on multi-domain fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below in conjunction with specific embodiments.

[0052] It should be noted that these embodiments are only used to illustrate the present invention rather than to limit the present invention. Simple improvements to the method based on the concept of the present invention all fall within the scope of protection claimed by the present invention.

[0053] See also Figure 1 , which is an image recolorization detection method based on multi-domain fusion, including:

[0054] S100: Acquire an image and perform size normalization processing on the image.

[0055] Before feature extraction, the input image needs to be normalized, that is, the input image is adjusted to a uniform size to ensure the consistency of the input data and the stability of feature extraction. Through the above preprocessing steps, the input image can achieve a standardized effect to ensure the accuracy and processing efficiency of subsequent feature extraction.

[0056] S200, converting the size-normalized image from the RGB space to the HSV color space to obtain an HSV image for color domain feature extraction.

[0057] Color domain feature extraction mainly analyzes the color distribution of the image to capture color anomalies caused by the recoloring operation. Since the recoloring process usually generates images in the RGB color space, some recoloring algorithms tend to follow the image features in the RGB space, resulting in tampering artifacts that are so subtle that it is impossible to extract tampering features from them. Since the recoloring algorithm pays less attention to features in other color spaces, these spaces can show color differences more clearly. This step converts the image from the RGB space to other color spaces (such as YCbCr, HSV, etc.), which can better analyze the color domain features of the image.

[0058] S300, extracting residual domain features of the image after size normalization through a residual feature extraction module.

[0059] Since the content of the recolored image is visually difficult to distinguish, it is not possible to capture the tampering traces from the color domain of the image alone to extract the distinguishing features. To solve this problem, the image is usually high-pass filtered to suppress the image content and obtain the residual of the image. In this way, the traces of tampering can be better captured. Based on this idea, the present invention designs a residual feature extraction module, which is divided into two sub-modules: a residual feature enhancement module and a residual feature extraction module based on the ResNet18 network.

[0060] The normalized image is passed through the residual feature enhancement module to extract the high-frequency information related to recoloring in the image using a high-pass filter kernel and enhanced, and then input into the ResNet18 residual feature extraction module. The high-level residual features are extracted through the ResNet18 residual feature extraction module to obtain the residual domain features of the image.

[0061] Specifically, the main purpose of the residual feature enhancement module is to use the high-pass filter kernel to extract high-frequency information related to recoloring in the image, enhance these features and then pass them as input to the subsequent network (ResNet18 residual feature extraction module). This module includes 3 The depthwise separable convolution is performed with a step size of 1. Each channel of the input image is convolved with a set of high-pass filter kernels, and the convolution results (a total of 9 image residual feature maps) are spliced ​​together as the input of the subsequent network layer for further shallow feature extraction. The initial high-pass filter kernel parameters used in this paper are:

[0062]

[0063] The above filter kernel parameters are designed to be learnable, that is, their elements can be adjusted by gradient descent during training to optimize the detection effect.

[0064] ResNet18 network-based residual feature extraction module / ResNet18 residual feature extraction module is used to further extract shallow residual features while ensuring that the size of the output feature map is consistent with the original RGB image. ResNet18 is a lighter convolutional neural network architecture with residual connections that can capture more complex features. Usually, the resolution of the output feature map of ResNet18 is lower than that of the input image. In order to facilitate the fusion of residual features, image color gamut features, and illumination map features. This step requires adjustments to ResNet18, as follows:

[0065] Convolution step size adjustment: In order to ensure that the spatial size of the feature map remains unchanged, all convolutional layers with a step size of 2 in the standard ResNet18 need to be adjusted to a step size of 1.

[0066] Reduce the number of channels: ResNet18 increases the number of channels in each residual block from 128 → 256 → 512 → 1024. In order to output 3 channels while maintaining the spatial size, it is necessary to insert a The convolution layer directly compresses the number of channels to 3. This operation does not affect the spatial resolution of the image, but only changes the number of channels. Convolution can learn linear combinations between channels, thereby mapping high-dimensional features to 3 channels, ensuring that the output residual feature size is By extracting features from the residual domain, we can effectively compensate for frequency anomalies that cannot be captured in the color domain features, so that image tampering behavior can be analyzed more comprehensively in multiple dimensions.

[0067] S400: Generate a light map from the image after size normalization.

[0068] The light map reflects the color and intensity of the light source for each pixel in the scene. The recoloring operation may cause inconsistent lighting information in different areas of the image, so the anomaly of the light map can be used as a significant feature of recoloring. The gray world hypothesis algorithm can be used to estimate the light source color and brightness distribution of the image. The specific steps are as follows:

[0069] Step 1: Calculate the global average of each color channel of the image.

[0070] First, the global averages of each color channel (R, G, B) of the input image are calculated. These global averages can help us infer the lighting conditions in the image.

[0071] The formula is as follows:

[0072]

[0073] in, is the number of pixels in the image, , , They are image The value of a pixel in R, G, and B channels.

[0074] Step 2: Estimate the light source color

[0075] According to the grayscale world hypothesis, under ideal conditions, the average values ​​of the three RGB channels should be equal. If there is a deviation in the channel average value, it means that the influence of the light source color (light color) causes this deviation. Therefore, the color of the light source can be estimated based on these deviations.

[0076] The light source color is estimated as the result of normalization to the mean value of each channel:

[0077]

[0078] in, , the purpose is to normalize the intensity of the light source color so that it does not exceed the average value of the largest channel.

[0079] Step 3: Generate a light map

[0080] Apply the light color estimate to each pixel to generate a light map. This is done by normalizing each pixel based on the light color estimate:

[0081]

[0082] in, Is the light map in pixels The value at , , is the R, G, B value of the input image at that pixel, , , are the three components of the inferred light source color.

[0083] After the illumination map is generated, the illumination intensity in different regions can be analyzed to detect whether there is abnormal light source information in the image. If the light source is inconsistent in different regions, it may mean that the image has been tampered with. In particular, in image recoloring detection, this inconsistency can be used as a basis for detecting recoloring operations.

[0084] S500, constructing a multi-domain fusion feature tensor according to the HSV image, residual domain features and illumination map of the acquired image.

[0085] After the above multi-domain feature extraction steps, for a size of (Image height is , the width is , the number of channels is ) and obtained RGB images of size The light map of size The HSV image of size is The illumination map, residual domain features, and HSV images are stacked by channel to form a feature tensor , the size is This feature tensor contains various information about the image, including color, lighting, and residual changes.

[0086] Multi-domain fusion feature tensor X f It will be divided into a series of non-overlapping patches, namely the image sequence. The position vector is the position vector of each patch in the original multi-domain fusion feature tensor X. f The relative position in .

[0087] S600 uses the Vision Transformer model to extract high, medium and low levels of image feature sequences, and concatenates the three levels of image feature sequences with their corresponding position vectors respectively, and inputs them into the encoder composed of the self-attention mechanism and the forward propagation network to extract the multi-scale deep features of the image.

[0088] For the extracted multi-domain fusion feature tensor (Size is ), the present invention adopts three ViT (Vision Transformer) models of different scales to perform multi-domain feature fusion and deep feature extraction.

[0089] Each ViT uses different patch sizes (4×4, 8×8, 16×16) to capture image features at different scales (high, medium, and low levels):

[0090] 4×4 Patch ViT: It consists of 6 layers of basic Transformer encoders, which are used to extract local fine high-level features and are suitable for capturing small-scale recoloring clues. The number of patches is , the dimension of each patch is 4×4×7=112, so the input sequence size is: .

[0091] 8×8 Patch ViT: It consists of 8 layers of basic Transformer encoders, extracting mid-level features and capturing mid-range recoloring signs. The number of patches is , the dimension of each patch is 8×8×7=448, so the input sequence size is: .

[0092] 16×16 Patch ViT: It consists of 12 layers of basic Transformer encoders, extracting global low-level features, suitable for capturing global tampering trends in images. The number of patches is , the dimension of each patch is: 16×16×7=1792, so the input sequence size is: .

[0093] The above three image sequences of different sizes are concatenated with their position vectors to make the Transformer encoder perceive the spatial information of the image. Then they are input into the encoder composed of the self-attention mechanism and the forward propagation network to extract the multi-scale deep features of the recolored image. The input sequence size is:

[0094] The above self-attention mechanism:

[0095] The role of the self-attention mechanism is to dynamically adjust the importance of different features by calculating the relationship between features, and then fuse multi-domain features. Transformer's self-attention mechanism includes three main processes:

[0096] Query, Key, and Value generation:

[0097] Transformer will input feature sequence Through three different linear transformations, three matrices are generated: Query matrix, Key matrix and Value matrix. The specific calculation formula is:

[0098] , ,

[0099] in, , , is the weight matrix that needs to be learned.

[0100] Calculate the attention weights:

[0101] By calculating the dot product of the Query matrix and the Key matrix and normalizing the result, we get the attention weight matrix , the formula is:

[0102]

[0103] in, is the dimension of the feature vector, and Softmax is used for normalization to ensure that the weight sums to 1.

[0104] Weighted feature fusion:

[0105] According to the attention weight matrix and Value matrix Perform weighted summation to obtain the fused feature representation, i.e., multi-scale deep features:

[0106]

[0107] The multi-scale deep features It is a deep feature representation generated by fusing the color domain, residual domain and illumination map features. , , Respectively Patch, Patch, Output of Patch ViT.

[0108] S700, based on the multi-scale deep features of the extracted image, a global feature map is generated by gradually upsampling and fusion.

[0109] After extracting the deep features of multi-domain fusion through multi-scale ViT, in order to accurately locate the tampered area of ​​recoloring, the output features of the three-scale ViT modules are gradually fused by gradually upsampling and fusion:

[0110] First upsampling: from Characteristics of Patch Encoder Output By using deconvolution to upsample to The features output by the Patch encoder have the same spatial resolution, and the upsampled features are Characteristics of Patch Encoder Output Feature fusion is performed by element-by-element addition.

[0111] Second upsampling: The features fused in the previous step are first subjected to the same deconvolution upsampling, and upsampled to The features output by the Patch encoder have the same spatial resolution, and the upsampled features are the same as the features output by the 4x4 Patch encoder. Perform element-wise addition fusion.

[0112] Final upsampling: The features fused in the previous step are again subjected to the same deconvolution upsampling to make their width and height the same as the original RGB image, and finally generate the fused global feature map , the size is .

[0113] S800, the global feature map is subjected to global pooling to fuse features at different positions into a global feature vector, and based on the obtained global feature vector, a classifier is used to perform a binary classification determination on whether the global feature vector has been recolored.

[0114] In order to make a global judgment on whether the image has been tampered with, the present invention uses a global classifier to fusion feature map Further processing is performed, and finally a classification model is used to detect whether the image has been recolored. The detection process of the present invention includes:

[0115] Global feature aggregation

[0116] Fusion feature map Aggregate into a fixed-dimensional vector representation through global pooling operations. For example, features at different locations are fused into a global feature vector through global average pooling to represent the comprehensive features of the entire image.

[0117] Classifier

[0118] The global feature vector is input into the classifier, which is usually one or more fully connected layers plus a Softmax activation function, to perform a binary classification (recolored / not recolored) on whether the image has been recolored. The formula of the classifier is:

[0119]

[0120] in, and is the weight matrix and bias term of the classifier, is the fusion feature, is the output class probability.

[0121] The image recoloring detection method based on multi-domain fusion of the present invention also includes S900, outputting a positioning map through a decoder according to the generated global feature map; and determining whether the pixel is tampered with by recoloring according to the size of the pixel value of each pixel in the positioning map.

[0122] The global feature map Input to In the decoder composed of convolutional layers, a recolored tampered localization map is generated , size is Each pixel value in the localization map represents the probability of whether the pixel has been tampered with by recoloring. The higher the value, the greater the possibility that the pixel has been tampered with.

[0123] The model training process of the present invention includes:

[0124] Data preparation: During the training phase, a training dataset containing original images and recolored images needs to be constructed. The preparation of the dataset includes the following steps:

[0125] Original image collection: Collect high-resolution, unedited images from public image databases (such as COCO, ImageNet, etc.) or your own image resources.

[0126] Recolored image generation: Use image recoloring algorithms or image editing tools to modify the color of the original image and generate a corresponding recolored version. These recoloring operations can include changing the overall tone of the image, local color adjustments, and applying color filters.

[0127] Ground Truth Mask: A binary mask (same size as the image) that indicates whether each pixel has been tampered with (0 means no tampering, 1 means tampering).

[0128] In order to enhance the robustness of the model, data augmentation methods can also be applied to the training data, such as rotation, scaling, flipping and other operations to simulate image changes in different scenarios.

[0129] Loss function design

[0130] For multi-task learning, it is necessary to design loss functions for classification tasks and localization tasks respectively and combine them. For classification tasks, the cross entropy loss function is used to measure the error between the model prediction and the true label. The optimization goal is to minimize the loss function:

[0131]

[0132] in, is the true label, is the predicted probability of the model.

[0133] For the positioning task, Dice loss is used to locate the pixel level of the tampered area. The formula is:

[0134]

[0135] in, is the true mask value, is the predicted probability value.

[0136] Total loss:

[0137]

[0138] in, and is the weight used to balance the importance of the two tasks.

[0139] Model initialization:

[0140] Initialize the ViT model: Set the initial weights for the ViT model. The ViT model can choose to start from a pre-trained Transformer model (such as a model pre-trained on ImageNet) to speed up training and improve initial performance.

[0141] Training parameter configuration The hyperparameters for training include:

[0142] Learning rate: The initial learning rate can be set to 0.001 and trained using the Adam optimizer.

[0143] Batch size: This can be set to 16 or 32, depending on the hardware resources.

[0144] Number of training rounds: Usually set to 100 to 200 epochs, depending on the size of the dataset and the convergence of the model.

[0145] Pre-filtering module: During the training process, residual domain features are extracted through high-pass filters, and the weights of these filters are learnable. In the early stage of training, the filter kernel is initialized and the weights of the filter are updated through the gradient descent algorithm, so that the model can dynamically adjust the filter parameters according to the input image to enhance the sensitivity to tampering traces.

[0146] Test result output and report generation

[0147] The test results are output in the form of images or reports:

[0148] Image form: Overlay the original image with the tampering location map that marks the recoloring area, and output a visual recoloring detection result to help users identify the parts of the image that may have been tampered with.

[0149] Report format: Generates a detailed detection report, including the classification results of the image, visual markings of the recoloring areas, and the possibility analysis of the recoloring operation.

[0150] See also Figure 2 , which is an image recoloring detection system based on multi-domain fusion, including an image acquisition and processing module 100, a color domain conversion module 200, a residual domain feature extraction module 300, an illumination map generation module 400, a multi-domain fusion feature tensor generation module 500, a multi-scale deep feature extraction module 600, a global feature map generation module 700 and a determination module 800.

[0151] The image acquisition and processing module 100 is used to acquire images and perform size normalization processing on the images.

[0152] In this module, before feature extraction, the input image needs to be normalized, that is, the input image is adjusted to a uniform size to ensure the consistency of the input data and the stability of feature extraction. Through the above preprocessing steps, the input image can achieve a standardized effect to ensure the accuracy and processing efficiency of subsequent feature extraction.

[0153] The color domain conversion module 200 is used to convert the image after size normalization from the RGB space to the HSV color space to obtain an HSV image for color domain feature extraction.

[0154] Color domain feature extraction mainly analyzes the color distribution of the image to capture color anomalies caused by the recoloring operation. Since the recoloring process usually generates images in the RGB color space, some recoloring algorithms tend to follow the image features in the RGB space, resulting in tampering artifacts that are so subtle that it is impossible to extract tampering features from them. Since the recoloring algorithm pays less attention to features in other color spaces, these spaces can show color differences more clearly. This step converts the image from the RGB space to other color spaces (such as YCbCr, HSV, etc.), which can better analyze the color domain features of the image.

[0155] The residual domain feature extraction module 300 is used to extract residual domain features of the image after the size normalization through the residual feature extraction module.

[0156] Since the content of the recolored image is visually difficult to distinguish, it is not possible to capture the tampering traces from the color domain of the image alone to extract the distinguishing features. To solve this problem, the image is usually high-pass filtered to suppress the image content and obtain the residual of the image. In this way, the traces of tampering can be better captured. Based on this idea, the present invention designs a residual feature extraction module, which is divided into two sub-modules: a residual feature enhancement module and a residual feature extraction module based on the ResNet18 network.

[0157] The normalized image is passed through the residual feature enhancement module to extract the high-frequency information related to recoloring in the image using a high-pass filter kernel and enhanced, and then input into the ResNet18 residual feature extraction module. The high-level residual features are extracted through the ResNet18 residual feature extraction module to obtain the residual domain features of the image.

[0158] Specifically, the main purpose of the residual feature enhancement module is to use the high-pass filter kernel to extract high-frequency information related to recoloring in the image, enhance these features and then pass them as input to the subsequent network. This module includes 3 The depthwise separable convolution is performed with a step size of 1. Each channel of the input image is convolved with a set of high-pass filter kernels, and the convolution results (a total of 9 image residual feature maps) are spliced ​​together as the input of the subsequent network layer for further shallow feature extraction. The initial high-pass filter kernel parameters used in this paper are:

[0159]

[0160] The above filter kernel parameters are designed to be learnable, that is, their elements can be adjusted by gradient descent during training to optimize the detection effect.

[0161] ResNet18 network-based residual feature extraction module / ResNet18 residual feature extraction module is used to further extract shallow residual features while ensuring that the size of the output feature map is consistent with the original RGB image. ResNet18 is a lighter convolutional neural network architecture with residual connections that can capture more complex features. Usually, the resolution of the output feature map of ResNet18 is lower than that of the input image. In order to facilitate the fusion of residual features, image color gamut features, and illumination map features. This step requires adjustments to ResNet18, as follows:

[0162] Convolution step size adjustment: In order to ensure that the spatial size of the feature map remains unchanged, all convolutional layers with a step size of 2 in the standard ResNet18 need to be adjusted to a step size of 1.

[0163] Reduce the number of channels: ResNet18 increases the number of channels in each residual block from 128 → 256 → 512 → 1024. In order to output 3 channels while maintaining the spatial size, it is necessary to insert a The convolution layer directly compresses the number of channels to 3. This operation does not affect the spatial resolution of the image, but only changes the number of channels. Convolution can learn linear combinations between channels, thereby mapping high-dimensional features to 3 channels, ensuring that the output residual feature size is By extracting features from the residual domain, we can effectively compensate for frequency anomalies that cannot be captured in the color domain features, so that image tampering behavior can be analyzed more comprehensively in multiple dimensions.

[0164] The illumination map generation module 400 is used to generate an illumination map from the image after size normalization.

[0165] The light map reflects the color and intensity of the light source for each pixel in the scene. The recoloring operation may cause inconsistent lighting information in different areas of the image, so the anomaly of the light map can be used as a significant feature of recoloring. The gray world hypothesis algorithm can be used to estimate the light source color and brightness distribution of the image. The specific steps are as follows:

[0166] Step 1: Calculate the global average of each color channel of the image.

[0167] First, the global averages of each color channel (R, G, B) of the input image are calculated. These global averages can help us infer the lighting conditions in the image.

[0168] The formula is as follows:

[0169]

[0170] in, is the number of pixels in the image, , , They are image The value of a pixel in R, G, and B channels.

[0171] Step 2: Estimate the light source color

[0172] According to the grayscale world hypothesis, under ideal conditions, the average values ​​of the three RGB channels should be equal. If there is a deviation in the channel average value, it means that the influence of the light source color (light color) causes this deviation. Therefore, the color of the light source can be estimated based on these deviations.

[0173] The light source color is estimated as the result of normalization to the mean value of each channel:

[0174]

[0175] in, , the purpose is to normalize the intensity of the light source color so that it does not exceed the average value of the largest channel.

[0176] Step 3: Generate a light map

[0177] Apply the light color estimate to each pixel to generate a light map. This is done by normalizing each pixel based on the light color estimate:

[0178]

[0179] in, Is the light map in pixels The value at , , is the R, G, B value of the input image at that pixel, , , are the three components of the inferred light source color.

[0180] After the illumination map is generated, the illumination intensity in different regions can be analyzed to detect whether there is abnormal light source information in the image. If the light source is inconsistent in different regions, it may mean that the image has been tampered with. In particular, in image recoloring detection, this inconsistency can be used as a basis for detecting recoloring operations.

[0181] The multi-domain fusion feature tensor generation module 500 is used to construct a multi-domain fusion feature tensor according to the HSV image, residual domain features and illumination map of the acquired image.

[0182] After the above multi-domain feature extraction, for a size of (Image height is , the width is , the number of channels is ) and obtained RGB images of size The light map of size The HSV image of size is The illumination map, residual domain features, and HSV images are stacked by channel to form a feature tensor , the size is This feature tensor contains various information about the image, including color, lighting, and residual changes.

[0183] The multi-scale deep feature extraction module 600 is used to extract high, medium and low level image feature sequences using the Vision Transformer model, and concatenate the three levels of image feature sequences with their corresponding position vectors respectively, and input them into the encoder composed of the self-attention mechanism and the forward propagation network to extract the multi-scale deep features of the image.

[0184] For the extracted multi-domain fusion feature tensor (Size is ), the present invention adopts three ViT (Vision Transformer) models of different scales to perform multi-domain feature fusion and deep feature extraction.

[0185] Each ViT uses different patch sizes (4×4, 8×8, 16×16) to capture image features at different scales (high, medium, and low levels):

[0186] 4×4 Patch ViT: It consists of 6 layers of basic Transformer encoders, which are used to extract local fine high-level features and are suitable for capturing small-scale recoloring clues. The number of patches is , the dimension of each patch is 4×4×7=112, so the input sequence size is: .

[0187] 8×8 Patch ViT: It consists of 8 layers of basic Transformer encoders, extracting mid-level features and capturing mid-range recoloring signs. The number of patches is , the dimension of each patch is 8×8×7=448, so the input sequence size is: .

[0188] 16×16 Patch ViT: It consists of 12 layers of basic Transformer encoders, extracting global low-level features, suitable for capturing global tampering trends in images. The number of patches is , the dimension of each patch is: 16×16×7=1792, so the input sequence size is: .

[0189] The above three image sequences of different sizes are concatenated with their position vectors to make the Transformer encoder perceive the spatial information of the image. Then they are input into the encoder composed of the self-attention mechanism and the forward propagation network to extract the multi-scale deep features of the recolored image.

[0190] The above self-attention mechanism:

[0191] The role of the self-attention mechanism is to dynamically adjust the importance of different features by calculating the relationship between features, and then fuse multi-domain features. Transformer's self-attention mechanism includes three main processes:

[0192] Query, Key, and Value generation:

[0193] Transformer will input feature sequence Through three different linear transformations, three matrices are generated: Query matrix, Key matrix and Value matrix. The specific calculation formula is:

[0194] , ,

[0195] in, , , is the weight matrix that needs to be learned.

[0196] Calculate the attention weights:

[0197] By calculating the dot product of the Query matrix and the Key matrix and normalizing the result, we get the attention weight matrix , the formula is:

[0198]

[0199] in, is the dimension of the feature vector, and Softmax is used for normalization to ensure that the weight sums to 1.

[0200] Weighted feature fusion:

[0201] According to the attention weight matrix and Value matrix Perform weighted summation to obtain the fused feature representation, i.e., multi-scale deep features:

[0202]

[0203] The multi-scale deep features It is a deep feature representation generated by fusing the color domain, residual domain and illumination map features. , , Respectively Patch, Patch, Output of Patch ViT.

[0204] The global feature map generation module 700 is used to generate a global feature map by gradually upsampling and fusion based on the multi-scale deep features of the extracted image.

[0205] After extracting the deep features of multi-domain fusion through multi-scale ViT, in order to accurately locate the tampered area of ​​recoloring, the output features of the three-scale ViT modules are gradually fused by gradually upsampling and fusion:

[0206] First upsampling: from Characteristics of Patch Encoder Output By using deconvolution to upsample to The features output by the Patch encoder have the same spatial resolution, and the upsampled features are Characteristics of Patch Encoder Output Feature fusion is performed by element-by-element addition.

[0207] Second upsampling: The features fused in the previous step are first subjected to the same deconvolution upsampling, and upsampled to The features output by the Patch encoder have the same spatial resolution, and the upsampled features are Characteristics of Patch Encoder Output Perform element-wise addition fusion.

[0208] Final upsampling: The features fused in the previous step are again subjected to the same deconvolution upsampling to make their width and height the same as the original RGB image, and finally generate the fused global feature map , the size is .

[0209] The determination module 800 is used to fuse the features at different positions of the global feature map into a global feature vector through global pooling, and based on the obtained global feature vector, use a classifier to perform a binary classification determination on whether recoloring has been performed.

[0210] In order to make a global judgment on whether the image has been tampered with, the present invention uses a global classifier to fusion feature map Further processing is performed, and finally a classification model is used to detect whether the image has been recolored. The detection process of the present invention includes:

[0211] Global feature aggregation

[0212] Fusion feature map Aggregate into a fixed-dimensional vector representation through global pooling operations. For example, features at different locations are fused into a global feature vector through global average pooling to represent the comprehensive features of the entire image.

[0213] Classifier

[0214] The global feature vector is input into the classifier, which is usually one or more fully connected layers plus a Softmax activation function, to perform a binary classification (recolored / not recolored) on whether the image has been recolored. The formula of the classifier is:

[0215]

[0216] in, and is the weight matrix and bias term of the classifier, is the fusion feature, is the output class probability.

[0217] The system of the present invention also includes a positioning module 900, which is used to output a positioning map through a decoder based on the generated global feature map; and determine whether the pixel is tampered with by recoloring based on the size of the pixel value of each pixel in the positioning map.

[0218] The global feature map Input to In the decoder composed of convolutional layers, a recolored tampered localization map is generated , size is Each pixel value in the localization map represents the probability of whether the pixel has been tampered with by recoloring. The higher the value, the greater the possibility that the pixel has been tampered with.

[0219] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the image recoloring detection method based on multi-domain fusion of the present invention are implemented.

[0220] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image recoloring detection method based on multi-domain fusion of the present invention.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. An image recoloring detection method based on multi-domain fusion, characterized in that: include: Acquire an image and normalize the image size; Convert the size-normalized image from the RGB space to the HSV color space to obtain an HSV image; The image after size normalization is used to extract residual domain features of the image through a residual feature extraction module; Generate a light map from the size-normalized image; Construct a multi-domain fusion feature tensor based on the HSV image, residual domain features and illumination map of the acquired image; The Vision Transformer model is used to extract high, medium and low level image feature sequences. The three levels of image feature sequences are concatenated with their corresponding position vectors and input into the encoder composed of the self-attention mechanism and the forward propagation network to extract the multi-scale deep features of the image. Based on the multi-scale deep features of the extracted image, a global feature map is generated by gradually upsampling and fusion; The global feature map is pooled to fuse features at different positions into a global feature vector. Based on the obtained global feature vector, a classifier is used to perform a binary classification judgment on whether it has been recolored.

2. The image recoloring detection method based on multi-domain fusion according to claim 1 is characterized in that: The method also includes outputting a positioning map through a decoder according to the generated global feature map; and determining whether the pixel is tampered with by recoloring according to the size of the pixel value of each pixel in the positioning map.

3. The image recoloring detection method based on multi-domain fusion according to claim 1, characterized in that: The step of extracting residual domain features of the image after the size normalization process by a residual feature extraction module comprises: The normalized image is passed through the residual feature enhancement module to extract the high-frequency information related to recoloring in the image using a high-pass filter kernel and enhanced, and then input into the ResNet18 residual feature extraction module. The high-level residual features are extracted through the ResNet18 residual feature extraction module to obtain the residual domain features of the image.

4. The image recoloring detection method based on multi-domain fusion according to claim 1, characterized in that: Generating a light map from the size-normalized image comprises: Calculate the global average value for each color channel of the image; The global average of each color channel is normalized to obtain the light source color estimate; The light map is generated by normalizing the light color estimate at each pixel and applying the light color estimate to each pixel.

5. The image recoloring detection method based on multi-domain fusion according to claim 1, characterized in that: The Vision Transformer model is used to extract high, medium and low level image feature sequences, including: A 4×4 Patch ViT is used to extract local fine high-level feature sequences. The 4×4 Patch ViT consists of a 6-layer basic Transformer encoder. The size of the input sequence is ; The 8×8 Patch ViT is used to extract the mid-level feature sequence. The 8×8 Patch ViT consists of 8 layers of basic Transformer encoders. The size of the input sequence is ; A 16×16 Patch ViT is used to extract the global low-level feature sequence. The 16×16 Patch ViT consists of a 12-layer basic Transformer encoder. The size of the input sequence is .

6. The image recoloring detection method based on multi-domain fusion according to claim 5 is characterized in that: The self-attention mechanism includes: The multi-domain fusion feature tensor is transformed through three different linear transformations to generate the Query matrix, Key matrix and Value matrix; The attention weight matrix is ​​obtained by calculating the dot product of the Query matrix and the Key matrix and normalizing them; According to the weighted summation of attention weights and Value matrix, multi-scale deep features are obtained.

7. The image recoloring detection method based on multi-domain fusion according to claim 6 is characterized in that: The method of generating a global feature map by stepwise upsampling and fusion based on the multi-scale deep features of the extracted image includes: The features output from the 16×16 Patch encoder are upsampled using deconvolution to the same spatial resolution as the features output from the 8×8 Patch encoder. The upsampled features are fused with the features output from the 8×8 Patch encoder by element-by-element addition. The features after feature fusion are upsampled by using deconvolution to the same spatial resolution as the features output by the 4×4 Patch encoder. The upsampled features are fused with the features output by the 4×4 Patch encoder by element-by-element addition. The features after feature fusion are upsampled using deconvolution to make their width and height the same as the acquired image, and finally a fused global feature map is generated.

8. An image recoloring detection system based on multi-domain fusion, characterized in that: It includes an image acquisition and processing module, a color domain conversion module, a residual domain feature extraction module, an illumination map generation module, a multi-domain fusion feature tensor generation module, a multi-scale deep feature extraction module, a global feature map generation module and a judgment module; The image acquisition and processing module is used to acquire images and perform size normalization processing on the images; The color domain conversion module is used to convert the image after size normalization from the RGB space to the HSV color space to obtain an HSV image; The residual domain feature extraction module is used to extract residual domain features of the image after the size normalization through the residual feature extraction module; The illumination map generation module is used to generate an illumination map from the image after size normalization; The multi-domain fusion feature tensor generation module is used to construct a multi-domain fusion feature tensor according to the HSV image, residual domain features and illumination map of the acquired image; The multi-scale deep feature extraction module is used to extract high, medium and low level image feature sequences using the Vision Transformer model, and concatenate the three levels of image feature sequences with their corresponding position vectors respectively, and input them into an encoder composed of a self-attention mechanism and a forward propagation network to extract multi-scale deep features of the image; The global feature map generation module is used to generate a global feature map by stepwise upsampling and fusion based on the multi-scale deep features of the extracted image; The determination module is used to fuse features at different positions of the global feature map into a global feature vector through global pooling, and based on the obtained global feature vector, use a classifier to perform a binary classification determination on whether the global feature map has been recolored.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image recoloring detection method based on multi-domain fusion as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image recoloring detection method based on multi-domain fusion as claimed in any one of claims 1 to 7 are implemented.

Images